Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 435929 reports (Page 7858 of 8719)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXX]
duration: 
30.247039811s
FAIL (2025-07-27 00:07:05 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.247060992s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX]
duration: 
30.000991283s
FAIL (2025-07-27 00:07:00 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001000911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................X.....................]
duration: 
1.836285196s
FAIL (2025-07-27 00:06:51 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.836308362s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..........]
duration: 
30.79533043s
FAIL (2025-07-27 00:06:49 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.795354036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [..XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.X]
duration: 
30.00191878s
FAIL (2025-07-27 00:06:30 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00194941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XX]
duration: 
30.239745298s
FAIL (2025-07-27 00:06:28 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.239761157s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXX]
duration: 
30.231596387s
FAIL (2025-07-27 00:06:27 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
s (needed at least 13 but got 10)
Jul 27 00:05:57.436782393 (60h30m7.159345433s) [upload=58029]: Starting upload
Jul 27 00:05:57.436841255 (60h30m7.159404295s) [upload=58029]: Done starting upload
Jul 27 00:05:57.436944794 (60h30m7.159507834s) [upload=58029]: Uploading segments...
Jul 27 00:05:57.437036960 (60h30m7.1596s) [upload=58029]: Got next segment. Inline: false
Jul 27 00:05:57.437078049 (60h30m7.159641089s) [upload=58029]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:05:57.437083178 (60h30m7.159646218s) [upload=58029]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:05:57.878627803 (60h30m7.601190843s) [seg_pos={0 0}, upload=58029]: Begin upload segment...
Jul 27 00:05:57.878668642 (60h30m7.601231682s) [seg_pos={0 0}, upload=58029]: Done begin upload segment.
Jul 27 00:05:57.878693149 (60h30m7.601256189s) [noise=true , address=45.91.116.14:4191    , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.878743705 (60h30m7.601306745s) [noise=true , address=45.91.116.7:4009     , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.879047837 (60h30m7.601610877s) [noise=true , address=77.90.31.17:4772     , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.879220728 (60h30m7.601783768s) [noise=true , address=104.36.82.7:4910     , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.879296172 (60h30m7.601859212s) [noise=true , address=104.36.82.11:4115    , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.879559025 (60h30m7.602122065s) [noise=true , address=45.91.116.5:4299     , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.879806168 (60h30m7.602369208s) [noise=true , address=104.36.82.7:4911     , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.880097045 (60h30m7.602660085s) [noise=true , address=104.36.82.8:4971     , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.880331633 (60h30m7.602894673s) [noise=true , address=45.91.116.9:4262     , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.880595328 (60h30m7.603158368s) [noise=true , address=77.90.31.18:4807     , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.880811973 (60h30m7.603375003s) [noise=true , address=104.36.82.12:4150    , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.881019630 (60h30m7.60358267s) [noise=true , address=45.91.116.10:4081    , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.881231736 (60h30m7.603794776s) [noise=true , address=77.90.31.15:4698     , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.881410217 (60h30m7.603973257s) [noise=true , address=77.90.31.14:4662     , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.881641800 (60h30m7.60420483s) [noise=true , address=104.36.82.6:4876     , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:57.881728897 (60h30m7.604291927s) [noise=true , address=45.91.116.13:4154    , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Uploading piece...
Jul 27 00:05:59.620668084 (60h30m9.343231124s) [upload=58029]: Next returned nil segment
Jul 27 00:05:59.620685557 (60h30m9.343248597s) [upload=58029]: Waiting for error group managing segments...
Jul 27 00:05:59.897369139 (60h30m9.619932179s) [noise=true , address=45.91.116.13:4154    , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:05:59.913229704 (60h30m9.635792744s) [noise=true , address=45.91.116.7:4009     , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:05:59.976379096 (60h30m9.698942136s) [noise=true , address=45.91.116.10:4081    , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:05:59.983423965 (60h30m9.705987005s) [noise=true , address=77.90.31.14:4662     , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:05:59.998286812 (60h30m9.720849852s) [noise=true , address=45.91.116.5:4299     , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:06:00.008836206 (60h30m9.731399236s) [noise=true , address=77.90.31.18:4807     , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:06:00.124184478 (60h30m9.846747508s) [noise=true , address=45.91.116.9:4262     , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:06:00.210814585 (60h30m9.933377625s) [noise=true , address=45.91.116.14:4191    , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil>
Jul 27 00:06:26.796472853 (60h30m36.519035893s) [noise=true , address=77.90.31.17:4772     , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796538539 (60h30m36.519101579s) [noise=true , address=104.36.82.8:4971     , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796576962 (60h30m36.519140002s) [noise=true , address=104.36.82.7:4911     , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:56848->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796622630 (60h30m36.51918567s) [noise=true , address=77.90.31.15:4698     , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:45116->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796661484 (60h30m36.519224524s) [noise=true , address=104.36.82.12:4150    , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796689097 (60h30m36.519252137s) [noise=true , address=104.36.82.6:4876     , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796710368 (60h30m36.519273408s) [noise=true , address=104.36.82.7:4910     , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115    , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments...
Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.231616634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................X....................]
duration: 
1.824645846s
FAIL (2025-07-27 00:06:22 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.82466946s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)
antiblip: 
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
1m0.246366359s
FAIL (2025-07-27 00:06:21 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
(60h29m32.51017865s) [noise=true , address=45.91.116.14:4206    , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.787943176 (60h29m32.510506217s) [noise=true , address=104.36.82.52:4091    , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788012329 (60h29m32.510575369s) [noise=true , address=104.36.82.19:4408    , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788241988 (60h29m32.510805028s) [noise=true , address=45.91.116.9:4277     , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788362539 (60h29m32.510925569s) [noise=true , address=79.127.205.246:4258  , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788489312 (60h29m32.511052342s) [noise=true , address=104.36.82.47:4530    , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788635791 (60h29m32.511198821s) [noise=true , address=79.127.205.234:4607  , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788801899 (60h29m32.511364939s) [noise=true , address=104.36.82.46:4511    , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.788915456 (60h29m32.511478496s) [noise=true , address=77.90.31.10:4523     , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789048411 (60h29m32.511611451s) [noise=true , address=64.49.13.138:4483    , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789151699 (60h29m32.511714739s) [noise=true , address=104.36.82.56:4231    , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789284383 (60h29m32.511847423s) [noise=true , address=104.36.82.53:4125    , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789392951 (60h29m32.511955991s) [noise=true , address=79.127.201.215:4345  , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789488373 (60h29m32.512051413s) [noise=true , address=77.90.31.19:4867     , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789637699 (60h29m32.512200739s) [noise=true , address=79.127.205.235:4657  , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789684658 (60h29m32.512247688s) [noise=true , address=104.36.82.54:4172    , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:22.789908547 (60h29m32.512471587s) [noise=true , address=79.127.220.101:4041  , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Uploading piece...
Jul 27 00:05:23.902236566 (60h29m33.624799596s) [upload=58019]: Next returned nil segment
Jul 27 00:05:23.902253699 (60h29m33.624816739s) [upload=58019]: Waiting for error group managing segments...
Jul 27 00:05:24.227516616 (60h29m33.950079656s) [noise=true , address=79.127.205.235:4657  , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:24.522516541 (60h29m34.245079581s) [noise=true , address=79.127.205.246:4258  , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:24.787777739 (60h29m34.510340779s) [noise=true , address=77.90.31.16:4759     , piece_id=OV6NCZRVNDFWVK35MASHEG2UFGQG33H6I2PTJJFLRFVPMSABD5ZA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:24.939622529 (60h29m34.662185569s) [noise=true , address=79.127.205.234:4607  , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:27.214858358 (60h29m36.937421398s) [noise=true , address=79.127.220.101:4041  , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:28.083413763 (60h29m37.805976803s) [noise=true , address=64.49.13.138:4483    , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:29.775391290 (60h29m39.49795432s) [noise=true , address=45.91.116.9:4277     , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000
Jul 27 00:05:31.705915531 (60h29m41.428478571s) [noise=true , address=77.90.31.19:4867     , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:34.195265036 (60h29m43.917828076s) [noise=true , address=45.91.116.14:4206    , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:05:56.693782311 (60h30m6.416345351s) [noise=true , address=79.127.201.215:4345  , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil>
Jul 27 00:06:21.700033712 (60h30m31.422596782s) [noise=true , address=104.36.82.52:4091    , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.700386977 (60h30m31.422950037s) [noise=true , address=77.90.31.10:4523     , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: read tcp 188.166.225.170:43628->77.90.31.10:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.700918505 (60h30m31.423481545s) [noise=true , address=104.36.82.46:4511    , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: rpc: dial tcp 104.36.82.46:4511: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.700947921 (60h30m31.423510961s) [noise=true , address=104.36.82.56:4231    , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.701018105 (60h30m31.423581165s) [noise=true , address=104.36.82.54:4172    , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 188.166.225.170:33396->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.701057230 (60h30m31.42362029s) [noise=true , address=104.36.82.19:4408    , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ, address:104.36.82.19:4408): protocol: rpc: dial tcp 104.36.82.19:4408: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.701066748 (60h30m31.423629788s) [noise=true , address=104.36.82.53:4125    , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:104.36.82.53:4125): protocol: rpc: dial tcp 104.36.82.53:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.701071377 (60h30m31.423634417s) [noise=true , address=104.36.82.47:4530    , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:06:21.701094822 (60h30m31.423657922s) [seg_pos={0 0}, upload=58019]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:06:21.944972146 (60h30m31.667535186s) [upload=58019]: Done uploading segments...
Jul 27 00:06:21.944996322 (60h30m31.667559362s) [upload=58019]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.246385647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [.....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.........]
duration: 
30.384339509s
FAIL (2025-07-27 00:06:04 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.38435079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.X]
duration: 
30.23505612s
FAIL (2025-07-27 00:05:52 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.235082952s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X........]
duration: 
38.482893493s
FAIL (2025-07-27 00:05:33 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.482914954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XX]
duration: 
30.241147367s
FAIL (2025-07-27 00:05:32 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
 closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments...
Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:05:02.560967093 (60h29m12.283530133s) [upload=58013]: Starting upload
Jul 27 00:05:02.561059250 (60h29m12.2836223s) [upload=58013]: Done starting upload
Jul 27 00:05:02.561208304 (60h29m12.283771364s) [upload=58013]: Uploading segments...
Jul 27 00:05:02.561391195 (60h29m12.283954245s) [upload=58013]: Got next segment. Inline: false
Jul 27 00:05:02.561445709 (60h29m12.284008749s) [upload=58013]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:05:02.561452602 (60h29m12.284015652s) [upload=58013]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:05:02.798729347 (60h29m12.521292407s) [seg_pos={0 0}, upload=58013]: Begin upload segment...
Jul 27 00:05:02.798805122 (60h29m12.521368162s) [seg_pos={0 0}, upload=58013]: Done begin upload segment.
Jul 27 00:05:02.798838295 (60h29m12.521401346s) [noise=true , address=77.90.31.13:4626     , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.798888512 (60h29m12.521451552s) [noise=true , address=77.90.31.15:4699     , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799021236 (60h29m12.521584276s) [noise=true , address=104.36.82.3:4765     , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799451228 (60h29m12.522014278s) [noise=true , address=45.91.116.10:4081    , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799464424 (60h29m12.522027464s) [noise=true , address=77.90.31.12:4590     , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799794074 (60h29m12.522357124s) [noise=true , address=45.91.116.7:4011     , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799850472 (60h29m12.522413512s) [noise=true , address=45.91.116.13:4154    , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800176917 (60h29m12.522739967s) [noise=true , address=45.91.116.14:4190    , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800307928 (60h29m12.522870978s) [noise=true , address=77.90.31.14:4663     , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800695489 (60h29m12.523258529s) [noise=true , address=104.36.82.12:4150    , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801003488 (60h29m12.523566528s) [noise=true , address=77.90.31.10:4518     , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801329863 (60h29m12.523892913s) [noise=true , address=77.90.31.18:4806     , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801678690 (60h29m12.52424174s) [noise=true , address=104.36.82.11:4113    , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.802064338 (60h29m12.524627388s) [noise=true , address=77.90.31.19:4842     , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.802411622 (60h29m12.524974672s) [noise=true , address=45.91.116.11:4118    , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.805333012 (60h29m12.527896052s) [noise=true , address=104.36.82.4:4801     , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:04.611541965 (60h29m14.334105005s) [upload=58013]: Next returned nil segment
Jul 27 00:05:04.611564408 (60h29m14.334127448s) [upload=58013]: Waiting for error group managing segments...
Jul 27 00:05:04.914940269 (60h29m14.637503309s) [noise=true , address=45.91.116.10:4081    , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.918629126 (60h29m14.641192166s) [noise=true , address=45.91.116.7:4011     , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.948737141 (60h29m14.671300181s) [noise=true , address=45.91.116.11:4118    , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.963852912 (60h29m14.686415952s) [noise=true , address=45.91.116.13:4154    , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.988745859 (60h29m14.711308929s) [noise=true , address=77.90.31.14:4663     , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.991103750 (60h29m14.71366679s) [noise=true , address=45.91.116.14:4190    , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.248956147 (60h29m15.971519197s) [noise=true , address=77.90.31.18:4806     , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.300358975 (60h29m16.022922016s) [noise=true , address=77.90.31.13:4626     , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.410168350 (60h29m16.13273139s) [noise=true , address=77.90.31.12:4590     , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.869083350 (60h29m16.59164639s) [noise=true , address=77.90.31.19:4842     , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:31.913473966 (60h29m41.636036996s) [noise=true , address=104.36.82.4:4801     , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913502281 (60h29m41.636065321s) [noise=true , address=77.90.31.10:4518     , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:43888->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913521568 (60h29m41.636084608s) [noise=true , address=104.36.82.3:4765     , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:56046->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913530945 (60h29m41.636093985s) [noise=true , address=77.90.31.15:4699     , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913536205 (60h29m41.636099245s) [noise=true , address=104.36.82.12:4150    , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913552858 (60h29m41.636115898s) [noise=true , address=104.36.82.11:4113    , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913571243 (60h29m41.636134283s) [seg_pos={0 0}, upload=58013]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:05:32.152667560 (60h29m41.8752306s) [upload=58013]: Done uploading segments...
Jul 27 00:05:32.152698479 (60h29m41.87526152s) [upload=58013]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.241166312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................X...................]
duration: 
2.65925967s
FAIL (2025-07-27 00:05:10 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 2.659289076s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)
antiblip: 
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
1m0.229973629s
FAIL (2025-07-27 00:05:08 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
g piece...
Jul 27 00:04:09.199649906 (60h28m18.922212946s) [noise=true , address=104.36.82.46:4502    , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200035684 (60h28m18.922598725s) [noise=true , address=79.127.220.101:4048  , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200040494 (60h28m18.922603534s) [noise=true , address=79.127.205.246:4281  , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200375855 (60h28m18.922938895s) [noise=true , address=104.36.82.55:4216    , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200793785 (60h28m18.923356825s) [noise=true , address=79.127.219.51:4210   , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201093718 (60h28m18.923656758s) [noise=true , address=104.36.82.54:4157    , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201396357 (60h28m18.923959397s) [noise=true , address=79.127.205.233:4646  , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201691182 (60h28m18.924254222s) [noise=true , address=104.36.82.2:4755     , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201979143 (60h28m18.924542183s) [noise=true , address=79.127.205.235:4666  , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202211828 (60h28m18.924774868s) [noise=true , address=79.127.219.47:4048   , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202447349 (60h28m18.925010389s) [noise=true , address=104.36.82.53:4138    , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202532802 (60h28m18.925095842s) [noise=true , address=79.127.205.245:4245  , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202798570 (60h28m18.92536161s) [noise=true , address=104.36.82.52:4117    , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203036606 (60h28m18.925599646s) [noise=true , address=79.127.219.38:4010   , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203222462 (60h28m18.925785502s) [noise=true , address=104.36.82.56:4253    , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203476147 (60h28m18.926039177s) [noise=true , address=104.36.82.7:4918     , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203826307 (60h28m18.926389337s) [noise=true , address=156.146.43.225:4448  , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:10.303379963 (60h28m20.025942993s) [upload=58000]: Next returned nil segment
Jul 27 00:04:10.303396825 (60h28m20.025959865s) [upload=58000]: Waiting for error group managing segments...
Jul 27 00:04:10.468485635 (60h28m20.191048675s) [noise=true , address=79.127.219.51:4210   , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000
Jul 27 00:04:10.618950635 (60h28m20.341513666s) [noise=true , address=79.127.205.246:4281  , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.623707968 (60h28m20.346271008s) [noise=true , address=79.127.205.245:4245  , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.628858482 (60h28m20.351421522s) [noise=true , address=79.127.205.233:4646  , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.632026083 (60h28m20.354589123s) [noise=true , address=79.127.205.235:4666  , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:11.122563874 (60h28m20.845126914s) [noise=true , address=79.127.219.38:4010   , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:11.445377285 (60h28m21.167940325s) [noise=true , address=79.127.220.101:4048  , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:12.775062857 (60h28m22.497625897s) [noise=true , address=79.127.219.47:4048   , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:12.926457887 (60h28m22.649020927s) [noise=true , address=156.146.43.225:4448  , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:21.769452468 (60h28m31.492015508s) [noise=true , address=64.49.13.138:4488    , piece_id=5XRHQYWD2BJXGHH7ABFJXDDXIVUS63LRSGRT2SXITYWSJ2WMP2FA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:05:08.019334096 (60h29m17.741897136s) [noise=true , address=104.36.82.56:4253    , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019386065 (60h29m17.741949105s) [noise=true , address=104.36.82.46:4502    , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019428085 (60h29m17.741991125s) [noise=true , address=104.36.82.53:4138    , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019461268 (60h29m17.742024309s) [noise=true , address=104.36.82.52:4117    , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: rpc: dial tcp 104.36.82.52:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019499592 (60h29m17.742062632s) [noise=true , address=104.36.82.55:4216    , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019535661 (60h29m17.742098701s) [noise=true , address=104.36.82.7:4918     , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: rpc: dial tcp 104.36.82.7:4918: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019575157 (60h29m17.742138197s) [noise=true , address=104.36.82.54:4157    , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019629892 (60h29m17.742192932s) [noise=true , address=104.36.82.2:4755     , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 188.166.225.170:40334->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019646233 (60h29m17.742209273s) [seg_pos={0 0}, upload=58000]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:05:08.246996305 (60h29m17.969559335s) [upload=58000]: Done uploading segments...
Jul 27 00:05:08.247022976 (60h29m17.969586016s) [upload=58000]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.229993076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [...XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.]
duration: 
5.944056939s
FAIL (2025-07-27 00:05:03 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.944068381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.X]
duration: 
30.245112591s
FAIL (2025-07-27 00:05:01 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
UZi3): protocol: expected piece hash; context canceled; EOF
Jul 27 00:03:40.824444038 (60h27m50.547007068s) [seg_pos={0 0}, upload=57993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 27 00:03:40.824514603 (60h27m50.547077643s) [upload=57993]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 27 00:03:40.824519062 (60h27m50.547082092s) [upload=57993]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 27 00:03:41.130112176 (60h27m50.852675216s) [upload=57993]: Done uploading segments...
Jul 27 00:03:41.130131623 (60h27m50.852694663s) [upload=57993]: Upload finished. err: <nil>
Jul 27 00:04:32.085896910 (60h28m41.80845995s) [upload=58008]: Starting upload
Jul 27 00:04:32.085951976 (60h28m41.808515016s) [upload=58008]: Done starting upload
Jul 27 00:04:32.086013413 (60h28m41.808576493s) [upload=58008]: Uploading segments...
Jul 27 00:04:32.086132050 (60h28m41.80869509s) [upload=58008]: Got next segment. Inline: false
Jul 27 00:04:32.086163139 (60h28m41.808726179s) [upload=58008]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:04:32.086168039 (60h28m41.808731069s) [upload=58008]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:04:32.316154546 (60h28m42.038717586s) [seg_pos={0 0}, upload=58008]: Begin upload segment...
Jul 27 00:04:32.316190194 (60h28m42.038753234s) [seg_pos={0 0}, upload=58008]: Done begin upload segment.
Jul 27 00:04:32.316211956 (60h28m42.038774986s) [noise=true , address=104.36.82.10:4079    , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.316550994 (60h28m42.039114034s) [noise=true , address=77.90.31.21:4877     , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.316820280 (60h28m42.03938332s) [noise=true , address=45.91.116.10:4083    , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317062914 (60h28m42.039625954s) [noise=true , address=77.90.31.10:4517     , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317476846 (60h28m42.040039886s) [noise=true , address=45.91.116.6:4974     , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317739519 (60h28m42.040302559s) [noise=true , address=77.90.31.14:4661     , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318258591 (60h28m42.040821631s) [noise=true , address=77.90.31.16:4733     , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318665460 (60h28m42.0412285s) [noise=true , address=45.91.116.4:4938     , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318967077 (60h28m42.041530117s) [noise=true , address=77.90.31.16:4734     , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319263524 (60h28m42.041826565s) [noise=true , address=45.91.116.8:4047     , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319569590 (60h28m42.04213263s) [noise=true , address=77.90.31.13:4626     , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319889312 (60h28m42.042452352s) [noise=true , address=104.36.82.9:4006     , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320214715 (60h28m42.042777755s) [noise=true , address=77.90.31.11:4555     , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320498017 (60h28m42.043061057s) [noise=true , address=77.90.31.15:4699     , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320812748 (60h28m42.043375788s) [noise=true , address=45.91.116.13:4155    , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.321152088 (60h28m42.043715138s) [noise=true , address=104.36.82.15:4257    , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:34.057760807 (60h28m43.780323847s) [upload=58008]: Next returned nil segment
Jul 27 00:04:34.057779792 (60h28m43.780342832s) [upload=58008]: Waiting for error group managing segments...
Jul 27 00:04:34.335101806 (60h28m44.057664846s) [noise=true , address=45.91.116.13:4155    , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.339370523 (60h28m44.061933563s) [noise=true , address=77.90.31.14:4661     , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.349741225 (60h28m44.072304255s) [noise=true , address=45.91.116.4:4938     , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.430079845 (60h28m44.152642875s) [noise=true , address=77.90.31.16:4734     , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.475555101 (60h28m44.198118141s) [noise=true , address=77.90.31.21:4877     , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.479221716 (60h28m44.201784746s) [noise=true , address=45.91.116.6:4974     , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.497190286 (60h28m44.219753326s) [noise=true , address=77.90.31.11:4555     , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.499780581 (60h28m44.222343621s) [noise=true , address=45.91.116.10:4083    , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.502073518 (60h28m44.224636558s) [noise=true , address=77.90.31.16:4733     , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.562247958 (60h28m44.284810988s) [noise=true , address=45.91.116.8:4047     , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:36.427408290 (60h28m46.14997133s) [noise=true , address=77.90.31.13:4626     , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:05:01.447187046 (60h29m11.169750086s) [noise=true , address=104.36.82.15:4257    , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:56162->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447238865 (60h29m11.169801905s) [noise=true , address=77.90.31.15:4699     , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447268642 (60h29m11.169831682s) [noise=true , address=104.36.82.10:4079    , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447297848 (60h29m11.169860898s) [noise=true , address=104.36.82.9:4006     , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:42962->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447363344 (60h29m11.169926394s) [noise=true , address=77.90.31.10:4517     , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58900->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments...
Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.245134211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X]
duration: 
30.001229295s
FAIL (2025-07-27 00:04:57 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001246707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.]
duration: 
7.109247136s
FAIL (2025-07-27 00:04:37 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 7.109270831s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................X..................]
duration: 
1.854811833s
FAIL (2025-07-27 00:04:30 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.854880083s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX]
duration: 
30.227430826s
FAIL (2025-07-27 00:04:19 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.227452569s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [.......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.......]
duration: 
10.157508953s
FAIL (2025-07-27 00:04:15 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.157530234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)
antiblip: 
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
1m0.234010391s
FAIL (2025-07-27 00:04:03 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
429914s) [noise=true , address=45.91.116.14:4206    , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296296305 (60h27m14.018859355s) [noise=true , address=104.36.82.40:4962    , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296448477 (60h27m14.019011567s) [noise=true , address=79.127.205.233:4633  , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296801893 (60h27m14.019364923s) [noise=true , address=79.127.219.38:4009   , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296916422 (60h27m14.019479462s) [noise=true , address=64.49.13.137:4502    , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297216586 (60h27m14.019779626s) [noise=true , address=79.127.205.234:4611  , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297386952 (60h27m14.019949992s) [noise=true , address=64.49.13.154:4927    , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297594509 (60h27m14.020157549s) [noise=true , address=77.90.31.10:4543     , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297733676 (60h27m14.020296716s) [noise=true , address=104.36.82.46:4504    , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297961742 (60h27m14.020524782s) [noise=true , address=104.36.82.54:4171    , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298019222 (60h27m14.020582262s) [noise=true , address=77.90.31.12:4620     , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298239854 (60h27m14.020802894s) [noise=true , address=79.127.205.245:4252  , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298338854 (60h27m14.020901894s) [noise=true , address=79.127.205.235:4669  , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298502326 (60h27m14.021065366s) [noise=true , address=104.36.82.47:4552    , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298706026 (60h27m14.021269066s) [noise=true , address=104.36.82.14:4245    , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298868206 (60h27m14.021431246s) [noise=true , address=79.127.201.215:4347  , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.299167178 (60h27m14.021730229s) [noise=true , address=104.36.82.56:4230    , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.299589586 (60h27m14.022152626s) [noise=true , address=104.36.82.12:4169    , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:05.385782620 (60h27m15.10834566s) [upload=57985]: Next returned nil segment
Jul 27 00:03:05.385828919 (60h27m15.108391969s) [upload=57985]: Waiting for error group managing segments...
Jul 27 00:03:05.701031808 (60h27m15.423594848s) [noise=true , address=79.127.205.234:4611  , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.701960976 (60h27m15.424524006s) [noise=true , address=79.127.205.235:4669  , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.722094938 (60h27m15.444657968s) [noise=true , address=79.127.205.245:4252  , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.743552713 (60h27m15.466115783s) [noise=true , address=79.127.205.233:4633  , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:06.278512358 (60h27m16.001075398s) [noise=true , address=79.127.219.38:4009   , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:08.340018352 (60h27m18.062581392s) [noise=true , address=77.90.31.12:4620     , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:11.565721846 (60h27m21.288284886s) [noise=true , address=64.49.13.137:4502    , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:13.988632217 (60h27m23.711195257s) [noise=true , address=64.49.13.154:4927    , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:18.683164753 (60h27m28.405727793s) [noise=true , address=45.91.116.14:4206    , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:30.503118221 (60h27m40.225681261s) [noise=true , address=79.127.201.215:4347  , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:04:03.190575544 (60h28m12.913138584s) [noise=true , address=104.36.82.54:4171    , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190656027 (60h28m12.913219067s) [noise=true , address=104.36.82.12:4169    , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190692197 (60h28m12.913255237s) [noise=true , address=104.36.82.56:4230    , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190745179 (60h28m12.913308219s) [noise=true , address=104.36.82.46:4504    , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:37536->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190772331 (60h28m12.913335371s) [noise=true , address=104.36.82.40:4962    , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190827717 (60h28m12.913390757s) [noise=true , address=77.90.31.10:4543     , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190887290 (60h28m12.91345033s) [noise=true , address=104.36.82.47:4552    , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190940382 (60h28m12.913503432s) [noise=true , address=104.36.82.14:4245    , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190980088 (60h28m12.913543158s) [seg_pos={0 0}, upload=57985]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 27 00:04:03.422740258 (60h28m13.145303289s) [upload=57985]: Done uploading segments...
Jul 27 00:04:03.422770667 (60h28m13.145333717s) [upload=57985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.234036131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X......]
duration: 
32.74714383s
FAIL (2025-07-27 00:03:56 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 32.747160172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......]
duration: 
6.248021118s
FAIL (2025-07-27 00:03:47 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.248043411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.]
duration: 
5.808010915s
FAIL (2025-07-27 00:03:43 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload
Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload
Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments...
Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false
Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment...
Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment.
Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771     , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083    , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974     , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010     , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877     , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733     , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226    , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520     , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517     , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257    , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113    , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048     , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154    , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937     , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663     , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975     , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment
Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments...
Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877     , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226    , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048     , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663     , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975     , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010     , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154    , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083    , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974     , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937     , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771     , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733     , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113    , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520     , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517     , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257    , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments...
Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 5.808034329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................X.................]
duration: 
1.842571085s
FAIL (2025-07-27 00:03:33 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.842596432s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX]
duration: 
30.237202557s
FAIL (2025-07-27 00:03:32 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload
Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload
Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments...
Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false
Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment...
Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment.
Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771     , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083    , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974     , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010     , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877     , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733     , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226    , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520     , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517     , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257    , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113    , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048     , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154    , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937     , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663     , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975     , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment
Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments...
Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877     , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226    , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048     , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663     , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975     , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010     , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154    , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083    , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974     , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937     , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771     , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733     , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113    , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520     , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517     , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257    , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments...
Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.237224419s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [.........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.....]
duration: 
50.911700599s
FAIL (2025-07-27 00:03:23 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.911720456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XX]
duration: 
30.236606689s
FAIL (2025-07-27 00:03:20 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.236625776s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................X................]
duration: 
2.011069868s
FAIL (2025-07-27 00:03:04 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 2.011098783s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)
antiblip: 
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
1m0.261749221s
FAIL (2025-07-27 00:02:52 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
[noise=true , address=104.36.82.52:4090    , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141114954 (60h26m2.863677994s) [noise=true , address=79.127.220.101:4048  , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141201730 (60h26m2.86376477s) [noise=true , address=104.36.82.47:4527    , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141378839 (60h26m2.863941869s) [noise=true , address=79.127.201.215:4356  , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141466387 (60h26m2.864029427s) [noise=true , address=104.36.82.46:4507    , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141624139 (60h26m2.864187179s) [noise=true , address=64.49.13.138:4482    , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141724240 (60h26m2.86428728s) [noise=true , address=79.127.205.245:4223  , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141888344 (60h26m2.864451384s) [noise=true , address=104.36.82.54:4176    , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142007212 (60h26m2.864570252s) [noise=true , address=79.127.219.47:4048   , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142133743 (60h26m2.864696783s) [noise=true , address=77.90.31.15:4714     , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142227984 (60h26m2.864791024s) [noise=true , address=104.36.82.56:4244    , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142377239 (60h26m2.864940279s) [noise=true , address=104.36.82.7:4931     , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142443786 (60h26m2.865006826s) [noise=true , address=104.36.82.55:4216    , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142668477 (60h26m2.865231517s) [noise=true , address=45.91.116.10:4093    , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142690899 (60h26m2.865253929s) [noise=true , address=79.127.205.233:4639  , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142868359 (60h26m2.865431399s) [noise=true , address=79.127.205.246:4286  , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:54.229399462 (60h26m3.951962502s) [upload=57968]: Next returned nil segment
Jul 27 00:01:54.229417798 (60h26m3.951980828s) [upload=57968]: Waiting for error group managing segments...
Jul 27 00:01:54.536628044 (60h26m4.259191084s) [noise=true , address=79.127.205.245:4223  , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.549230276 (60h26m4.271793316s) [noise=true , address=79.127.205.233:4639  , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.582728527 (60h26m4.305291567s) [noise=true , address=79.127.205.235:4685  , piece_id=3WBPT3C3LCIK2XY25Y5M7FO2CVM4MHZ4DUGBDVSS64EXDI766M3Q, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.609006844 (60h26m4.331569884s) [noise=true , address=79.127.205.246:4286  , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:55.637242753 (60h26m5.359805783s) [noise=true , address=64.49.13.138:4482    , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000
Jul 27 00:01:55.706028969 (60h26m5.428592009s) [noise=true , address=45.91.116.10:4093    , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:57.495722163 (60h26m7.218285203s) [noise=true , address=79.127.219.47:4048   , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:58.749704002 (60h26m8.472267042s) [noise=true , address=79.127.220.101:4048  , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:58.762131119 (60h26m8.484694189s) [noise=true , address=79.127.201.215:4356  , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:02:52.042028487 (60h27m1.764591517s) [noise=true , address=104.36.82.46:4507    , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042073464 (60h27m1.764636494s) [noise=true , address=104.36.82.39:4730    , piece_id=FDJH44ZA4Z6LXLNLQFLR5Y2B365AKY5J3X4G3M44SBI5PRMZI5FQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:104.36.82.39:4730): protocol: rpc: dial tcp 104.36.82.39:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042109052 (60h27m1.764672092s) [noise=true , address=104.36.82.52:4090    , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 188.166.225.170:32790->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042141123 (60h27m1.764704163s) [noise=true , address=104.36.82.54:4176    , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042285480 (60h27m1.76484852s) [noise=true , address=104.36.82.7:4931     , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL, address:104.36.82.7:4931): protocol: rpc: dial tcp 104.36.82.7:4931: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042300318 (60h27m1.764863358s) [noise=true , address=104.36.82.47:4527    , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042359180 (60h27m1.76492222s) [noise=true , address=104.36.82.55:4216    , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042365232 (60h27m1.764928272s) [noise=true , address=77.90.31.15:4714     , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: read tcp 188.166.225.170:58426->77.90.31.15:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042394899 (60h27m1.764957939s) [noise=true , address=104.36.82.56:4244    , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 188.166.225.170:53112->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042423023 (60h27m1.764986063s) [seg_pos={0 0}, upload=57968]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Jul 27 00:02:52.301418572 (60h27m2.023981632s) [upload=57968]: Done uploading segments...
Jul 27 00:02:52.301447227 (60h27m2.024010267s) [upload=57968]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 1m0.261765342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.X]
duration: 
30.234335876s
FAIL (2025-07-27 00:02:50 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.234357208s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXX]
duration: 
30.234483068s
FAIL (2025-07-27 00:02:39 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
26260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:02:09.926623655 (60h26m19.649186695s) [upload=57970]: Starting upload
Jul 27 00:02:09.926674673 (60h26m19.649237863s) [upload=57970]: Done starting upload
Jul 27 00:02:09.926868904 (60h26m19.649431934s) [upload=57970]: Uploading segments...
Jul 27 00:02:09.926902659 (60h26m19.649465689s) [upload=57970]: Got next segment. Inline: false
Jul 27 00:02:09.926938417 (60h26m19.649501457s) [upload=57970]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:02:09.926942434 (60h26m19.649505474s) [upload=57970]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:02:10.159693471 (60h26m19.882256511s) [seg_pos={0 0}, upload=57970]: Begin upload segment...
Jul 27 00:02:10.159735501 (60h26m19.882298541s) [seg_pos={0 0}, upload=57970]: Done begin upload segment.
Jul 27 00:02:10.159757503 (60h26m19.882320543s) [noise=true , address=45.91.116.7:4010     , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.159819051 (60h26m19.882382091s) [noise=true , address=77.90.31.21:4877     , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160178649 (60h26m19.882741679s) [noise=true , address=77.90.31.12:4590     , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160383892 (60h26m19.882946922s) [noise=true , address=77.90.31.15:4699     , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160457392 (60h26m19.883020432s) [noise=true , address=104.36.82.12:4149    , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160718953 (60h26m19.883281993s) [noise=true , address=45.91.116.14:4189    , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160747237 (60h26m19.883310277s) [noise=true , address=77.90.31.13:4627     , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161021091 (60h26m19.883584131s) [noise=true , address=104.36.82.14:4221    , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161057069 (60h26m19.883620109s) [noise=true , address=104.36.82.3:4765     , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161365109 (60h26m19.883928139s) [noise=true , address=45.91.116.13:4154    , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161371051 (60h26m19.883934091s) [noise=true , address=77.90.31.18:4805     , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161619686 (60h26m19.884182887s) [noise=true , address=104.36.82.16:4293    , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161651557 (60h26m19.884214597s) [noise=true , address=104.36.82.8:4969     , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161875946 (60h26m19.884438976s) [noise=true , address=104.36.82.7:4909     , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161921764 (60h26m19.884484794s) [noise=true , address=45.91.116.8:4048     , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.162120615 (60h26m19.884683645s) [noise=true , address=45.91.116.10:4083    , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:11.908387111 (60h26m21.630950151s) [upload=57970]: Next returned nil segment
Jul 27 00:02:11.908404243 (60h26m21.630967283s) [upload=57970]: Waiting for error group managing segments...
Jul 27 00:02:12.184893462 (60h26m21.907456502s) [noise=true , address=45.91.116.7:4010     , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.185162627 (60h26m21.907725657s) [noise=true , address=77.90.31.18:4805     , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.230461587 (60h26m21.953024617s) [noise=true , address=45.91.116.10:4083    , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.256849564 (60h26m21.979412604s) [noise=true , address=45.91.116.13:4154    , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.262586270 (60h26m21.98514931s) [noise=true , address=77.90.31.13:4627     , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.283764439 (60h26m22.006327479s) [noise=true , address=77.90.31.21:4877     , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.353431912 (60h26m22.075994953s) [noise=true , address=45.91.116.8:4048     , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.482312036 (60h26m22.204875076s) [noise=true , address=45.91.116.14:4189    , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:13.616478367 (60h26m23.339041407s) [noise=true , address=77.90.31.12:4590     , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:39.279759003 (60h26m49.002322084s) [noise=true , address=104.36.82.16:4293    , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279833485 (60h26m49.002396525s) [noise=true , address=104.36.82.3:4765     , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:38402->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279870246 (60h26m49.002433286s) [noise=true , address=104.36.82.14:4221    , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:44646->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279916685 (60h26m49.002479725s) [noise=true , address=104.36.82.7:4909     , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279955159 (60h26m49.002518199s) [noise=true , address=77.90.31.15:4699     , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279987791 (60h26m49.002550821s) [noise=true , address=104.36.82.8:4969     , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41792->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280021024 (60h26m49.002584074s) [noise=true , address=104.36.82.12:4149    , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:33538->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.234525259s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X.....]
duration: 
6.329550432s
FAIL (2025-07-27 00:02:34 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.329570659s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [..........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X....]
duration: 
32.728698643s
FAIL (2025-07-27 00:02:27 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 32.728720753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.]
duration: 
7.023299469s
FAIL (2025-07-27 00:02:09 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 7.023323927s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXX]
duration: 
30.232953879s
FAIL (2025-07-27 00:02:05 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
n, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments...
Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:01:35.539401218 (60h25m45.261964258s) [upload=57962]: Starting upload
Jul 27 00:01:35.539463527 (60h25m45.262026557s) [upload=57962]: Done starting upload
Jul 27 00:01:35.539640455 (60h25m45.262203496s) [upload=57962]: Uploading segments...
Jul 27 00:01:35.539671956 (60h25m45.262234996s) [upload=57962]: Got next segment. Inline: false
Jul 27 00:01:35.539692415 (60h25m45.262255445s) [upload=57962]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:01:35.539696553 (60h25m45.262259593s) [upload=57962]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:01:35.773032700 (60h25m45.49559574s) [seg_pos={0 0}, upload=57962]: Begin upload segment...
Jul 27 00:01:35.773073748 (60h25m45.495636778s) [seg_pos={0 0}, upload=57962]: Done begin upload segment.
Jul 27 00:01:35.773095660 (60h25m45.49565869s) [noise=true , address=77.90.31.21:4878     , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.773470618 (60h25m45.496033658s) [noise=true , address=45.91.116.7:4011     , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.773974211 (60h25m45.496537251s) [noise=true , address=77.90.31.19:4842     , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774008596 (60h25m45.496571676s) [noise=true , address=104.36.82.11:4114    , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774410496 (60h25m45.496973536s) [noise=true , address=77.90.31.16:4733     , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774513844 (60h25m45.497076884s) [noise=true , address=45.91.116.13:4153    , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774727582 (60h25m45.497290612s) [noise=true , address=45.91.116.13:4154    , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774883190 (60h25m45.49744623s) [noise=true , address=45.91.116.4:4937     , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775210867 (60h25m45.497773907s) [noise=true , address=45.91.116.14:4191    , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775268106 (60h25m45.497831146s) [noise=true , address=104.36.82.12:4150    , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775591756 (60h25m45.498154796s) [noise=true , address=45.91.116.6:4974     , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775923420 (60h25m45.49848646s) [noise=true , address=45.91.116.9:4263     , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776273600 (60h25m45.49883664s) [noise=true , address=77.90.31.15:4699     , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776579666 (60h25m45.499142706s) [noise=true , address=77.90.31.10:4518     , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776864541 (60h25m45.499427581s) [noise=true , address=77.90.31.12:4592     , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.777213028 (60h25m45.499776068s) [noise=true , address=104.36.82.3:4765     , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:37.516632604 (60h25m47.239195644s) [upload=57962]: Next returned nil segment
Jul 27 00:01:37.516644707 (60h25m47.239207747s) [upload=57962]: Waiting for error group managing segments...
Jul 27 00:01:37.793047892 (60h25m47.515610932s) [noise=true , address=45.91.116.13:4154    , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.877611314 (60h25m47.600174354s) [noise=true , address=45.91.116.14:4191    , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.886850622 (60h25m47.609413662s) [noise=true , address=45.91.116.7:4011     , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.887828463 (60h25m47.610391503s) [noise=true , address=45.91.116.6:4974     , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.894226173 (60h25m47.616789213s) [noise=true , address=77.90.31.21:4878     , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.954015056 (60h25m47.676578096s) [noise=true , address=45.91.116.4:4937     , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.954674427 (60h25m47.677237468s) [noise=true , address=45.91.116.13:4153    , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.968328110 (60h25m47.690891161s) [noise=true , address=45.91.116.9:4263     , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.090232961 (60h25m48.812796001s) [noise=true , address=77.90.31.12:4592     , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.245905354 (60h25m48.968468394s) [noise=true , address=77.90.31.15:4699     , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.909345256 (60h25m49.631908297s) [noise=true , address=77.90.31.19:4842     , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:02:04.895836786 (60h26m14.618399826s) [noise=true , address=104.36.82.3:4765     , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:57196->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.895965743 (60h26m14.618528783s) [noise=true , address=77.90.31.10:4518     , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.895998005 (60h26m14.618561045s) [noise=true , address=104.36.82.11:4114    , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:37496->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896051637 (60h26m14.618614678s) [noise=true , address=77.90.31.16:4733     , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45436->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896131230 (60h26m14.61869428s) [noise=true , address=104.36.82.12:4150    , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896156248 (60h26m14.618719288s) [seg_pos={0 0}, upload=57962]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:02:05.126235353 (60h26m14.848798393s) [upload=57962]: Done uploading segments...
Jul 27 00:02:05.126260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.232968676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X]
duration: 
30.235725345s
FAIL (2025-07-27 00:01:56 UTC) 
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.235739561s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)
antiblip: 
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
1m0.232792664s
FAIL (2025-07-27 00:01:51 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
, upload=57952]: Uploading piece...
Jul 27 00:00:52.690310647 (60h25m2.412873687s) [noise=true , address=45.91.116.14:4194    , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690629588 (60h25m2.413192628s) [noise=true , address=79.127.219.47:4048   , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690755619 (60h25m2.413318659s) [noise=true , address=104.36.82.47:4526    , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690902470 (60h25m2.41346551s) [noise=true , address=79.127.219.38:4018   , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691053178 (60h25m2.413616218s) [noise=true , address=104.36.82.52:4095    , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691357762 (60h25m2.413920802s) [noise=true , address=45.91.116.14:4206    , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691444227 (60h25m2.414007267s) [noise=true , address=104.36.82.53:4149    , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691650000 (60h25m2.41421304s) [noise=true , address=79.127.205.234:4608  , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691722409 (60h25m2.414285449s) [noise=true , address=79.127.220.97:4361   , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691963541 (60h25m2.414526571s) [noise=true , address=77.90.31.10:4540     , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692069282 (60h25m2.414632322s) [noise=true , address=79.127.201.215:4352  , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692265959 (60h25m2.414828999s) [noise=true , address=64.49.13.138:4482    , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692443869 (60h25m2.415006899s) [noise=true , address=156.146.43.225:4448  , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692516518 (60h25m2.415079559s) [noise=true , address=64.49.13.138:4483    , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692758370 (60h25m2.41532141s) [noise=true , address=104.36.82.12:4168    , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692778279 (60h25m2.415341309s) [noise=true , address=104.36.82.46:4499    , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692999933 (60h25m2.415562973s) [noise=true , address=104.36.82.49:4998    , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:53.707783843 (60h25m3.430346883s) [noise=true , address=79.127.219.47:4048   , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000
Jul 27 00:00:53.808335285 (60h25m3.530898325s) [upload=57952]: Next returned nil segment
Jul 27 00:00:53.808353410 (60h25m3.53091645s) [upload=57952]: Waiting for error group managing segments...
Jul 27 00:00:53.894991431 (60h25m3.617554461s) [noise=true , address=79.127.220.97:4361   , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000
Jul 27 00:00:54.294741272 (60h25m4.017304302s) [noise=true , address=79.127.205.234:4608  , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.372553368 (60h25m4.095116398s) [noise=true , address=64.49.13.138:4482    , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.551991192 (60h25m4.274554232s) [noise=true , address=79.127.219.38:4018   , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.786651822 (60h25m4.509214872s) [noise=true , address=45.91.116.14:4194    , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:56.389526811 (60h25m6.112089851s) [noise=true , address=45.91.116.14:4206    , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:57.469684549 (60h25m7.192247589s) [noise=true , address=156.146.43.225:4448  , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:57.904513196 (60h25m7.627076236s) [noise=true , address=79.127.201.214:4321  , piece_id=FQBU6OOVFDLBJCM44XHHWTGH3BOHMTC2XOBJH3L6DNOQAAFIQHOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:59.255459436 (60h25m8.978022476s) [noise=true , address=64.49.13.138:4483    , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:01:20.907151736 (60h25m30.629714776s) [noise=true , address=79.127.201.215:4352  , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:01:51.585590532 (60h26m1.308153573s) [noise=true , address=104.36.82.53:4149    , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585652411 (60h26m1.308215451s) [noise=true , address=77.90.31.10:4540     , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 188.166.225.170:37456->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585885096 (60h26m1.308448136s) [noise=true , address=104.36.82.46:4499    , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:35230->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585951222 (60h26m1.308514263s) [noise=true , address=104.36.82.52:4095    , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585973626 (60h26m1.308536686s) [noise=true , address=104.36.82.12:4168    , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586015997 (60h26m1.308579037s) [noise=true , address=104.36.82.49:4998    , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:104.36.82.49:4998): protocol: rpc: dial tcp 104.36.82.49:4998: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586060892 (60h26m1.308623932s) [noise=true , address=104.36.82.47:4526    , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:33012->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586097172 (60h26m1.308660243s) [seg_pos={0 0}, upload=57952]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:01:51.816141159 (60h26m1.538704189s) [upload=57952]: Done uploading segments...
Jul 27 00:01:51.816172810 (60h26m1.53873585s) [upload=57952]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.232819476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [............................................X...............]
duration: 
1.862142079s
FAIL (2025-07-27 00:01:51 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.862169621s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X....]
duration: 
7.628504267s
FAIL (2025-07-27 00:01:44 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.628519625s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [...........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...]
duration: 
22.886072806s
FAIL (2025-07-27 00:01:41 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.886084397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX]
duration: 
30.23914945s
FAIL (2025-07-27 00:01:34 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
ent: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments...
Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:01:05.077717119 (60h25m14.800280159s) [upload=57953]: Starting upload
Jul 27 00:01:05.077788786 (60h25m14.800351827s) [upload=57953]: Done starting upload
Jul 27 00:01:05.077967438 (60h25m14.800530478s) [upload=57953]: Uploading segments...
Jul 27 00:01:05.078083480 (60h25m14.80064652s) [upload=57953]: Got next segment. Inline: false
Jul 27 00:01:05.078125841 (60h25m14.800688881s) [upload=57953]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:01:05.078131832 (60h25m14.800694872s) [upload=57953]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:01:05.310276219 (60h25m15.03283926s) [seg_pos={0 0}, upload=57953]: Begin upload segment...
Jul 27 00:01:05.310332166 (60h25m15.032895207s) [seg_pos={0 0}, upload=57953]: Done begin upload segment.
Jul 27 00:01:05.310344830 (60h25m15.03290787s) [noise=true , address=77.90.31.12:4590     , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310361623 (60h25m15.032924663s) [noise=true , address=104.36.82.10:4078    , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310472134 (60h25m15.033035174s) [noise=true , address=45.91.116.11:4118    , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310905342 (60h25m15.033468382s) [noise=true , address=104.36.82.16:4293    , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311221878 (60h25m15.033784918s) [noise=true , address=104.36.82.8:4969     , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311550277 (60h25m15.034113327s) [noise=true , address=45.91.116.9:4261     , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311893673 (60h25m15.034456713s) [noise=true , address=45.91.116.10:4081    , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312251939 (60h25m15.034814989s) [noise=true , address=77.90.31.15:4699     , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312583363 (60h25m15.035146403s) [noise=true , address=45.91.116.6:4975     , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312902724 (60h25m15.035465774s) [noise=true , address=104.36.82.2:4729     , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.313247313 (60h25m15.035810353s) [noise=true , address=45.91.116.8:4045     , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.313579468 (60h25m15.036142508s) [noise=true , address=45.91.116.13:4154    , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.314389629 (60h25m15.036952679s) [noise=true , address=104.36.82.7:4910     , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.315145515 (60h25m15.037708565s) [noise=true , address=77.90.31.19:4841     , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.316289474 (60h25m15.038852524s) [noise=true , address=77.90.31.18:4806     , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.316765935 (60h25m15.039328985s) [noise=true , address=45.91.116.12:4227    , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:07.052152676 (60h25m16.774715716s) [upload=57953]: Next returned nil segment
Jul 27 00:01:07.052172283 (60h25m16.774735323s) [upload=57953]: Waiting for error group managing segments...
Jul 27 00:01:07.346109275 (60h25m17.068672315s) [noise=true , address=45.91.116.9:4261     , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.346622458 (60h25m17.069185498s) [noise=true , address=45.91.116.8:4045     , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.420735295 (60h25m17.143298325s) [noise=true , address=45.91.116.10:4081    , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.439976829 (60h25m17.162539869s) [noise=true , address=45.91.116.12:4227    , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.445103078 (60h25m17.167666118s) [noise=true , address=45.91.116.13:4154    , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.455448041 (60h25m17.178011081s) [noise=true , address=77.90.31.12:4590     , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.480860661 (60h25m17.203423732s) [noise=true , address=45.91.116.11:4118    , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.494123216 (60h25m17.216686246s) [noise=true , address=45.91.116.6:4975     , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.505061495 (60h25m17.227624535s) [noise=true , address=77.90.31.18:4806     , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:08.692327849 (60h25m18.414890889s) [noise=true , address=77.90.31.19:4841     , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:08.870774407 (60h25m18.593337457s) [noise=true , address=77.90.31.15:4699     , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:34.428248528 (60h25m44.150811578s) [noise=true , address=104.36.82.10:4078    , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:38342->104.36.82.10:4078: use of closed network connection; conn closed: op
Jul 27 00:01:34.428315156 (60h25m44.150878226s) [noise=true , address=104.36.82.2:4729     , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:49100->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:34.428374339 (60h25m44.150937379s) [noise=true , address=104.36.82.8:4969     , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428446246 (60h25m44.151009296s) [noise=true , address=104.36.82.16:4293    , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:34.428586766 (60h25m44.151149836s) [noise=true , address=104.36.82.7:4910     , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments...
Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.239168005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
1.949046689s
FAIL (2025-07-27 00:01:21 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.949078792s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
18 out of 60 is failed (threshold=3) [............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..]
duration: 
29.413031147s
FAIL (2025-07-27 00:01:18 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.413046005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXX]
duration: 
30.235163086s
FAIL (2025-07-27 00:01:04 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments...
Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:00:34.625856171 (60h24m44.348419211s) [upload=57946]: Starting upload
Jul 27 00:00:34.625958327 (60h24m44.348521407s) [upload=57946]: Done starting upload
Jul 27 00:00:34.626040784 (60h24m44.348603824s) [upload=57946]: Uploading segments...
Jul 27 00:00:34.626264823 (60h24m44.348827863s) [upload=57946]: Got next segment. Inline: false
Jul 27 00:00:34.626311623 (60h24m44.348874663s) [upload=57946]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:00:34.626317774 (60h24m44.348880844s) [upload=57946]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:00:34.865594065 (60h24m44.588157106s) [seg_pos={0 0}, upload=57946]: Begin upload segment...
Jul 27 00:00:34.865647087 (60h24m44.588210127s) [seg_pos={0 0}, upload=57946]: Done begin upload segment.
Jul 27 00:00:34.865670111 (60h24m44.588233141s) [noise=true , address=45.91.116.14:4190    , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.865708626 (60h24m44.588271666s) [noise=true , address=45.91.116.13:4153    , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866075106 (60h24m44.588638146s) [noise=true , address=104.36.82.15:4259    , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866256885 (60h24m44.588819925s) [noise=true , address=45.91.116.8:4045     , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866404937 (60h24m44.588967978s) [noise=true , address=77.90.31.18:4805     , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866648443 (60h24m44.589211484s) [noise=true , address=45.91.116.5:4297     , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866667950 (60h24m44.58923099s) [noise=true , address=45.91.116.7:4009     , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866930773 (60h24m44.589493813s) [noise=true , address=45.91.116.12:4226    , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867221260 (60h24m44.58978429s) [noise=true , address=104.36.82.10:4078    , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867547112 (60h24m44.590110152s) [noise=true , address=104.36.82.2:4731     , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867777634 (60h24m44.590340674s) [noise=true , address=45.91.116.6:4973     , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868010669 (60h24m44.590573699s) [noise=true , address=77.90.31.13:4627     , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868350270 (60h24m44.59091331s) [noise=true , address=45.91.116.13:4154    , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868577584 (60h24m44.591140624s) [noise=true , address=45.91.116.11:4118    , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868844886 (60h24m44.591407926s) [noise=true , address=104.36.82.14:4222    , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.869181019 (60h24m44.591744049s) [noise=true , address=104.36.82.8:4970     , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:36.571647152 (60h24m46.294210192s) [upload=57946]: Next returned nil segment
Jul 27 00:00:36.571683281 (60h24m46.294246331s) [upload=57946]: Waiting for error group managing segments...
Jul 27 00:00:36.869992438 (60h24m46.592555478s) [noise=true , address=45.91.116.11:4118    , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.887108205 (60h24m46.609671245s) [noise=true , address=45.91.116.14:4190    , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.907036524 (60h24m46.629599574s) [noise=true , address=45.91.116.8:4045     , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.958513032 (60h24m46.681076072s) [noise=true , address=45.91.116.7:4009     , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.973207827 (60h24m46.695770867s) [noise=true , address=77.90.31.13:4627     , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.974133909 (60h24m46.696696949s) [noise=true , address=45.91.116.13:4154    , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.977135692 (60h24m46.699698732s) [noise=true , address=45.91.116.13:4153    , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.978503389 (60h24m46.701066429s) [noise=true , address=77.90.31.18:4805     , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.016312929 (60h24m46.738875969s) [noise=true , address=45.91.116.5:4297     , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.021475726 (60h24m46.744038766s) [noise=true , address=45.91.116.12:4226    , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.036167487 (60h24m46.758730527s) [noise=true , address=45.91.116.6:4973     , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:01:03.968881072 (60h25m13.691444112s) [noise=true , address=104.36.82.14:4222    , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968922361 (60h25m13.691485401s) [noise=true , address=104.36.82.15:4259    , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:53216->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968928994 (60h25m13.691492034s) [noise=true , address=104.36.82.8:4970     , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:37916->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968952910 (60h25m13.69151595s) [noise=true , address=104.36.82.10:4078    , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968971746 (60h25m13.691534786s) [noise=true , address=104.36.82.2:4731     , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments...
Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.235193273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X...]
duration: 
12.246817958s
FAIL (2025-07-27 00:01:03 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.246844077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
1.99403249s
FAIL (2025-07-27 00:00:50 UTC) 
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.99405258s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
19 out of 60 is failed (threshold=3) [X............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.]
duration: 
50.489387343s
FAIL (2025-07-27 00:00:48 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.489408945s core uplink us-select-1 us1-placement-20