Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.505686453s FAIL (2025-10-08 15:41:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 7.50571624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.452706222s FAIL (2025-10-08 15:40:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.452739407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.144050183s FAIL (2025-10-08 15:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 8.144071442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.107580617s FAIL (2025-10-08 15:39:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.107618429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.173035054s FAIL (2025-10-08 15:38:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.173067616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.443779959s FAIL (2025-10-08 15:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.443824283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 23.47364203s FAIL (2025-10-08 15:37:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 23.47367378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 6.761402509s FAIL (2025-10-08 15:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.761436144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 7.884919948s FAIL (2025-10-08 15:35:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 7.884954686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 7.345083676s FAIL (2025-10-08 15:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 7.345117328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................................X...............] duration: 18.454732189s FAIL (2025-10-08 15:34:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 18.454757576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................................X..............] duration: 5.914484868s FAIL (2025-10-08 15:33:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 5.914518312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................................X.............] duration: 11.80737633s FAIL (2025-10-08 15:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 11.807405255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................................X............] duration: 6.216881407s FAIL (2025-10-08 15:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.216912886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................................X...........] duration: 6.032287109s FAIL (2025-10-08 15:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.032330301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................................X..........] duration: 12.276014109s FAIL (2025-10-08 15:29:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 12.27604553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................................X.........] duration: 6.375060681s FAIL (2025-10-08 15:28:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.375180139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................................X........] duration: 8.323464601s FAIL (2025-10-08 15:27:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 8.323502633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................................X.......] duration: 17.285541222s FAIL (2025-10-08 15:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 17.285578472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......................................X......] duration: 6.22741882s FAIL (2025-10-08 15:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.227450009s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.143804248s FAIL (2025-10-08 15:25:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.143828194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.............X.......................................X.....] duration: 20.268099092s FAIL (2025-10-08 15:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 20.268130623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............X.......................................X....] duration: 18.889710361s FAIL (2025-10-08 15:24:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 18.889739766s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.350009336s FAIL (2025-10-08 15:24:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.350060844s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.177973639s FAIL (2025-10-08 15:24:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178000338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.............X.......................................X...] duration: 6.435033452s FAIL (2025-10-08 15:24:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 6.435155265s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.310592682s FAIL (2025-10-08 15:23:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.310619976s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.............X.......................................X..] duration: 7.399193824s FAIL (2025-10-08 15:23:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 7.399238799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X.......................................X.] duration: 22.988903774s FAIL (2025-10-08 15:22:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 22.988923504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X.......................................X] duration: 30.004553742s FAIL (2025-10-08 15:22:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 15:21:50.448893384 (361h50m35.910985867s) [upload=318311]: Starting upload Oct 8 15:21:50.448962275 (361h50m35.911054748s) [upload=318311]: Done starting upload Oct 8 15:21:50.449096040 (361h50m35.911188513s) [upload=318311]: Uploading segments... Oct 8 15:21:50.449204025 (361h50m35.911296539s) [upload=318311]: Got next segment. Inline: false Oct 8 15:21:50.449255924 (361h50m35.911348397s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 15:21:50.449261445 (361h50m35.911353918s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 15:21:50.748327130 (361h50m36.210419623s) [seg_pos={0 0}, upload=318311]: Begin upload segment... Oct 8 15:21:50.748394879 (361h50m36.210487362s) [seg_pos={0 0}, upload=318311]: Done begin upload segment. Oct 8 15:21:50.748444383 (361h50m36.210536856s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.748479139 (361h50m36.210571622s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749005191 (361h50m36.211097674s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.749429940 (361h50m36.211522413s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.750775672 (361h50m36.212868156s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751143833 (361h50m36.213236306s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.751470626 (361h50m36.213563109s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752025563 (361h50m36.214118046s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.752571111 (361h50m36.214663594s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753475984 (361h50m36.215568467s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.753896416 (361h50m36.215988899s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.757036925 (361h50m36.219129398s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.759148846 (361h50m36.221241329s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.760963883 (361h50m36.223056366s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.763314579 (361h50m36.225407052s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:50.767116980 (361h50m36.229209453s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Uploading piece... Oct 8 15:21:52.424534517 (361h50m37.886627s) [upload=318311]: Next returned nil segment Oct 8 15:21:52.424559945 (361h50m37.886652428s) [upload=318311]: Waiting for error group managing segments... Oct 8 15:21:52.732748593 (361h50m38.194841076s) [noise=true , address=145.79.227.5:4222 , piece_id=G3K4K5DW5FA4XSVHBI3PT4HX3AZOACADDBXO2AJ5FZJYVNVS453A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748502581 (361h50m38.210595064s) [noise=true , address=145.79.227.16:4909 , piece_id=H2XAAKW6Q4UVPVUQUMKQLM4POU7MHIQUUTYHDW6GSFDN246YN5KQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748932280 (361h50m38.211024763s) [noise=true , address=145.79.227.13:4802 , piece_id=ETCUGZ2CSOW2HPZPVSN3SYWOMRJC6XQLHDP6PDOEKTMZLGWOM6VA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.748971264 (361h50m38.211063747s) [noise=true , address=145.79.227.7:4295 , piece_id=F3PN2ESBP5NA2RYC7U2LTUXNJ42YLJLDSISY4SCDQRAXA25ZDZKA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.749881527 (361h50m38.211974s) [noise=true , address=145.79.227.2:4114 , piece_id=XDQCBO2NCKBV62ZZJ6Y5NAZPMVZHATXQBOHKCCIKJAUZ4KD7UHIA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.762259598 (361h50m38.224352071s) [noise=true , address=145.79.227.15:4873 , piece_id=344I2AIRKGMN5QKV57IXBMMTHP44NWCEKKAON5XCA5VN2T2NEMSQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.772902284 (361h50m38.234994767s) [noise=true , address=145.79.227.14:4837 , piece_id=KQJVOYQ2DZ2TFGUEXRHI5RA5C7KQA5Y2XERBOUTLHMBHTTVYGJWA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.776567043 (361h50m38.238659526s) [noise=true , address=145.79.227.18:4007 , piece_id=I5LJW6TVMPMZXTQQ7RTBHSE2MDYJUDNO3AUZVVPXWQEXUJYYRAPA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:52.793645083 (361h50m38.255737566s) [noise=true , address=145.79.227.3:4151 , piece_id=U5NRBUUA65LOMNFHFGYDVACZQBDZF4OI4DVWGMQUXT7QGDJRCMDQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.019065511 (361h50m38.481157994s) [noise=true , address=77.90.31.10:4518 , piece_id=HUWCIPIIHSK2J6CKLJ3KJNCDLQSZCABXDXEAZZA25Q43KBHXPPPQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.133343612 (361h50m38.595436085s) [noise=true , address=77.90.31.21:4878 , piece_id=5RPIXPDZ7RKBVZACVUAZSUU3ZPS7CNRLM3PGNXNZRVNIC3QT6LJQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.140247248 (361h50m38.602339731s) [noise=true , address=45.91.116.12:4225 , piece_id=DQKDUQJNPPBAOZZP5RKSW7J2UXH4USUZMMPYVELJFRTMLW42MQTQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195875937 (361h50m38.65796842s) [noise=true , address=77.90.31.19:4841 , piece_id=NFJTIXGRTPHGSGZ6CWBRAZ5CLWWN4Y6IY37R5NNOWIRDGLY7PK2A, seg_pos={0 0}, upload=318311]: Done uploading piece. err: <nil> Oct 8 15:21:53.195927795 (361h50m38.658020268s) [seg_pos={0 0}, upload=318311]: Segment reached optimal threshold of 13 pieces. Oct 8 15:21:53.196191938 (361h50m38.658284411s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196214392 (361h50m38.658306865s) [noise=true , address=45.91.116.6:4975 , piece_id=47QPMA6XBY5UZWIJAEAGLIL2IQI2ZLD3JWNYE5LU6JQ6C5A6PRVQ, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196253596 (361h50m38.658346079s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196259838 (361h50m38.658352311s) [noise=true , address=45.91.116.8:4048 , piece_id=2GWDV55LKPFPZCVESPUBYMWCIKO5NGQH3SGIHM3QRKMPSREMCXSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196271319 (361h50m38.658363812s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 8 15:21:53.196284605 (361h50m38.658377088s) [noise=true , address=77.90.31.19:4843 , piece_id=LF72OCCPL3QEQUEB6FMMLIOAF6XG33CLYLJFEQFUY7QNIDNZKMSA, seg_pos={0 0}, upload=318311]: Long tail cancellation due to optimal threshold Oct 8 15:21:53.196313270 (361h50m38.658405743s) [seg_pos={0 0}, upload=318311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 15:21:53.196414242 (361h50m38.658506725s) [upload=318311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 15:21:53.196421426 (361h50m38.658513909s) [upload=318311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 15:21:53.657826861 (361h50m39.119919344s) [upload=318311]: Done uploading segments... Oct 8 15:21:53.657846048 (361h50m39.119938531s) [upload=318311]: Upload finished. err: <nil> Duration: 30.004645986s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.152028591s FAIL (2025-10-08 15:22:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.152103183s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.230186109s FAIL (2025-10-08 15:21:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.230217709s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.419046409s FAIL (2025-10-08 15:20:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.419147653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............X.......................................] duration: 6.669660024s FAIL (2025-10-08 15:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 6.669690594s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.308964127s FAIL (2025-10-08 15:20:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.308996889s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............X......................................] duration: 12.501911432s FAIL (2025-10-08 15:19:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 12.501937091s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.144464917s FAIL (2025-10-08 15:19:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.144489102s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.............X.....................................] duration: 6.15281374s FAIL (2025-10-08 15:19:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 6.152849638s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.278041828s FAIL (2025-10-08 15:18:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.278066035s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............X....................................] duration: 12.552211452s FAIL (2025-10-08 15:18:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 12.552242963s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.569114776s FAIL (2025-10-08 15:18:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.56914845s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............X...................................] duration: 6.786176143s FAIL (2025-10-08 15:18:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 6.786211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.2710543s FAIL (2025-10-08 15:17:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.271086993s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............X..................................] duration: 6.920913465s FAIL (2025-10-08 15:17:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 6.920951256s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.324141462s FAIL (2025-10-08 15:16:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.324167791s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............X.................................] duration: 10.975335984s FAIL (2025-10-08 15:16:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 10.975378516s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.172414626s FAIL (2025-10-08 15:15:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.172444744s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............X................................] duration: 15.746422958s FAIL (2025-10-08 15:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 15.746460318s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.354198758s FAIL (2025-10-08 15:15:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.354239795s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.............X...............................] duration: 6.534756203s FAIL (2025-10-08 15:14:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 14:52:36.796602051 (361h21m22.258694534s) [upload=317902]: Starting upload Oct 8 14:52:36.796682915 (361h21m22.258775388s) [upload=317902]: Done starting upload Oct 8 14:52:36.796934323 (361h21m22.259026796s) [upload=317902]: Uploading segments... Oct 8 14:52:36.796987385 (361h21m22.259079858s) [upload=317902]: Got next segment. Inline: false Oct 8 14:52:36.797025317 (361h21m22.25911779s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 14:52:36.797030116 (361h21m22.259122589s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 14:52:37.059225067 (361h21m22.52131755s) [seg_pos={0 0}, upload=317902]: Begin upload segment... Oct 8 14:52:37.059310911 (361h21m22.521403424s) [seg_pos={0 0}, upload=317902]: Done begin upload segment. Oct 8 14:52:37.059356548 (361h21m22.521449051s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059371245 (361h21m22.521463728s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059454575 (361h21m22.521547048s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059947673 (361h21m22.522040156s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.059955668 (361h21m22.522048141s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060341423 (361h21m22.522433896s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060350841 (361h21m22.522443315s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060724132 (361h21m22.522816615s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.060843580 (361h21m22.522936063s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061091381 (361h21m22.523183864s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061967890 (361h21m22.524060373s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.061983891 (361h21m22.524076364s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.062850271 (361h21m22.524942744s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.064218787 (361h21m22.52631127s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.066060373 (361h21m22.528152866s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:37.067865280 (361h21m22.529957763s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Uploading piece... Oct 8 14:52:38.770518615 (361h21m24.232611099s) [upload=317902]: Next returned nil segment Oct 8 14:52:38.770550977 (361h21m24.23264347s) [upload=317902]: Waiting for error group managing segments... Oct 8 14:52:39.051646290 (361h21m24.513738773s) [noise=true , address=145.79.227.5:4223 , piece_id=QYNVBPZHWQVT3OBJ5N7MJN7ZFGDBIBEVLKD3LFQG3YW5ID3PMYVA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.052809475 (361h21m24.514901958s) [noise=true , address=145.79.227.13:4804 , piece_id=KZYFZJODHGSZPN3VK4HORPZKFOAQAZOPWGLCJIGCXHCFD22MMLRQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.084066214 (361h21m24.546158697s) [noise=true , address=145.79.227.6:4258 , piece_id=G5HZQ3EUYV67KHBQRTGOF3IWDOLNILUL3Y6VAV6KUF27CEGUCXPQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.187427074 (361h21m24.649519547s) [noise=true , address=145.79.227.7:4295 , piece_id=3IBCNTZQLQMQWU4VDA74Q6OCRICEN3VOCUWW4BOJGIYUOLUNIENQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.211499472 (361h21m24.673591955s) [noise=true , address=145.79.227.16:4909 , piece_id=CVSUOKSTBSM662GXTE7ZFTWYPLQRXHSTS5FGU56OQ6X5AMBJUFSQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.311765490 (361h21m24.773857963s) [noise=true , address=145.79.227.14:4839 , piece_id=ZDDBSAMLKCH4LMUPSPGQIC5WCW5YP7LJQV4YOHYCJYJZIWB5FUOQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.377096212 (361h21m24.839188685s) [noise=true , address=77.90.31.21:4879 , piece_id=SZ3R7QTAR4CES7G4MBM332FABBWXQHB3DK4FDMVXBRU6AFFEVZAA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.397152403 (361h21m24.859244876s) [noise=true , address=145.79.227.4:4187 , piece_id=NSTE2B6OODDO4VFLYFJ3O6EJU6K5V3QDDU2OS6F44B7EPKRU4RDA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401145998 (361h21m24.863238481s) [noise=true , address=45.91.116.10:4081 , piece_id=WAVHCX33K7H3T4F32T4SD5PJGR4VZ575YJV2P5AXG6HUVWJQAKBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.401687679 (361h21m24.863780152s) [noise=true , address=45.91.116.13:4153 , piece_id=X7NUJU7UUWYZOWIPKDA4YDUA6WHFJUK4JE6S7J6V22JMZRPFPZ5A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.477718257 (361h21m24.93981074s) [noise=true , address=77.90.31.12:4589 , piece_id=NV53JLCYFQB5QIJZQTLUTQ6BSCMQJF6BRDPVYSI3JR5ZECL3SUGQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.486554964 (361h21m24.948647447s) [noise=true , address=77.90.31.18:4807 , piece_id=EZPVTUL6XXI6SKRGHJML2TVDIFPRN3CPQJ5OZJJXVED2547RGMBA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500786323 (361h21m24.962878796s) [noise=true , address=45.91.116.8:4047 , piece_id=P7ZL4GOKDBMEL4C6BPGKAPCBIW4OKG4HVC7OXRZOP5OGY67WEBCQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: <nil> Oct 8 14:52:39.500807514 (361h21m24.962899987s) [seg_pos={0 0}, upload=317902]: Segment reached optimal threshold of 13 pieces. Oct 8 14:52:39.500949594 (361h21m24.963042077s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.500959252 (361h21m24.963051725s) [noise=true , address=45.91.116.14:4191 , piece_id=VW5GMST5UV5K6SQBXQLZJJ5SKMEP3DOWIYH52C5OOZSM5QGSZH7A, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501077789 (361h21m24.963170282s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->77.90.31.13:4627: use of closed network connection Oct 8 14:52:39.501177307 (361h21m24.96326982s) [noise=true , address=77.90.31.13:4627 , piece_id=H5AUHNHY6S3JKVK6ITDR3QL7JE3PZRHBQ3D2DTQ6ZHRUSTUSJHFQ, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501191725 (361h21m24.963284208s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 8 14:52:39.501214017 (361h21m24.9633065s) [noise=true , address=77.90.31.11:4555 , piece_id=E52XPS5YPSK2BSW2G3BQC32OLMDIR2T3UY7WAYDWF42CWVV2CDCA, seg_pos={0 0}, upload=317902]: Long tail cancellation due to optimal threshold Oct 8 14:52:39.501265626 (361h21m24.963358109s) [seg_pos={0 0}, upload=317902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 14:52:39.501363252 (361h21m24.963455735s) [upload=317902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 14:52:39.501369203 (361h21m24.963461686s) [upload=317902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 14:52:39.821205997 (361h21m25.28329848s) [upload=317902]: Done uploading segments... Oct 8 14:52:39.821231496 (361h21m25.283323979s) [upload=317902]: Upload finished. err: <nil> Duration: 6.534794024s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)