Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232173144s FAIL (2025-07-26 23:45:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.232194814s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236220514s FAIL (2025-07-26 23:45:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== oad=57697]: Uploading piece... Jul 26 23:44:15.865777023 (60h8m25.588340053s) [noise=true , address=79.127.219.51:4212 , piece_id=VNMVPLDEITA2UXSOV5KKZDLWVHAONJVWMTA6OHP3ZGSEFDS56KWA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.865810007 (60h8m25.588373047s) [noise=true , address=104.36.82.52:4099 , piece_id=M36SDTUWEJ56462XRHIDZ2KT4HNEGUQJMBRNPBF23AM5GENU3XBQ, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866092978 (60h8m25.588656008s) [noise=true , address=77.90.31.13:4659 , piece_id=5R6BPI2FZZBGIVZSYWGB5WLBYCWJAV7Z7KXVJKAZ3YIG7W4O4VSQ, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866101204 (60h8m25.588664244s) [noise=true , address=104.36.82.56:4263 , piece_id=KP7NCM3YPEODH7FQPZULNAHLY2GEAMYWTMHCSTA5SQCHOKLNZGDA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866340823 (60h8m25.588903863s) [noise=true , address=79.127.205.234:4616 , piece_id=4OKU7MM43OYLBOCEQMLQGWBGPVZOUASZBZWICLGZLGC74LG3EJQA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866380658 (60h8m25.588943698s) [noise=true , address=77.90.31.10:4528 , piece_id=U5ZBTFMYANJ34HWP2LQZSITDT6A4ZTG34LFU3L4ERMS6SFIA6PAA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866596942 (60h8m25.589159982s) [noise=true , address=45.91.116.9:4273 , piece_id=4M5SIG3PKFJTX6QOI2N7XIQVOT4FONFZPMU25BIPTSHMGWWXUMEQ, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866659081 (60h8m25.589222121s) [noise=true , address=79.127.205.246:4282 , piece_id=F46L3UD3IVORPP6KJCM3BKWIEZXYHL2X2GW4PBU5C7W5GPZGG45Q, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866850076 (60h8m25.589413116s) [noise=true , address=104.36.82.47:4541 , piece_id=ZYY3HJO3M5WIOMIIMTSDMGVQ3QNWUH52V7XHZLYHMBS7DIJBXWQA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.866934287 (60h8m25.589497327s) [noise=true , address=104.36.82.46:4506 , piece_id=6A3NVN5EP6LXPYNIY3AS26YG4MQCG5XFSS7SESLNJNJCWRIS75DA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.867090487 (60h8m25.589653527s) [noise=true , address=104.36.82.13:4194 , piece_id=HIOCKLJCZTSHZEM27XUNCXFBXE2G6AROTOAXTZAHDIRRFJJ6MG6A, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.867215055 (60h8m25.589778095s) [noise=true , address=79.127.219.38:4001 , piece_id=RSAG7YMQEXPSQCEYRVYTVGIILQEHDBIM4MURO3QETJEX7XV4BKPA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.867316789 (60h8m25.589879819s) [noise=true , address=104.36.82.54:4176 , piece_id=NWBOBIRRB7BTGD42SJLJXE5MPQSEQGDCVTQV2GOBW4MWORSJNBIA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.867431379 (60h8m25.589994419s) [noise=true , address=77.90.31.16:4766 , piece_id=CZRX5R553LQRF337VHGAXVOQQ6QZX54DLCXI75R44MZKGBEQ4LYA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.867671909 (60h8m25.590234949s) [noise=true , address=77.90.31.18:4813 , piece_id=T5V5SU4MVZ42YAQMSAHNZSVL2WEDJLUOVW3OUNUEHBFSIH3ONCXA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:15.869561184 (60h8m25.592124224s) [noise=true , address=79.127.220.98:4421 , piece_id=XGBGMKOVALM3RSURVBLJ7HV5MTADJA7ZR644P2OU265E2YXSDOIA, seg_pos={0 0}, upload=57697]: Uploading piece... Jul 26 23:44:16.812028316 (60h8m26.534591356s) [noise=true , address=79.127.220.101:4048 , piece_id=NSC2SCKLVOTYTCIO7ML7KN7GXU65M2IARLUXNHIQMFHOQGHLHZWQ, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:44:16.973788342 (60h8m26.696351383s) [upload=57697]: Next returned nil segment Jul 26 23:44:16.973855420 (60h8m26.69641847s) [upload=57697]: Waiting for error group managing segments... Jul 26 23:44:17.267779866 (60h8m26.990342906s) [noise=true , address=79.127.205.234:4616 , piece_id=4OKU7MM43OYLBOCEQMLQGWBGPVZOUASZBZWICLGZLGC74LG3EJQA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:17.292891893 (60h8m27.015454943s) [noise=true , address=79.127.205.246:4282 , piece_id=F46L3UD3IVORPP6KJCM3BKWIEZXYHL2X2GW4PBU5C7W5GPZGG45Q, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:17.879769675 (60h8m27.602332715s) [noise=true , address=77.90.31.18:4813 , piece_id=T5V5SU4MVZ42YAQMSAHNZSVL2WEDJLUOVW3OUNUEHBFSIH3ONCXA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:17.967861011 (60h8m27.690424051s) [noise=true , address=79.127.219.38:4001 , piece_id=RSAG7YMQEXPSQCEYRVYTVGIILQEHDBIM4MURO3QETJEX7XV4BKPA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:18.599594568 (60h8m28.322157598s) [noise=true , address=79.127.220.98:4421 , piece_id=XGBGMKOVALM3RSURVBLJ7HV5MTADJA7ZR644P2OU265E2YXSDOIA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:20.506125813 (60h8m30.228688853s) [noise=true , address=79.127.219.51:4212 , piece_id=VNMVPLDEITA2UXSOV5KKZDLWVHAONJVWMTA6OHP3ZGSEFDS56KWA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:44:28.478185379 (60h8m38.200748419s) [noise=true , address=45.91.116.9:4273 , piece_id=4M5SIG3PKFJTX6QOI2N7XIQVOT4FONFZPMU25BIPTSHMGWWXUMEQ, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:45:07.948552219 (60h9m17.671115259s) [noise=true , address=77.90.31.13:4659 , piece_id=5R6BPI2FZZBGIVZSYWGB5WLBYCWJAV7Z7KXVJKAZ3YIG7W4O4VSQ, seg_pos={0 0}, upload=57697]: Done uploading piece. err: <nil> Jul 26 23:45:14.774938665 (60h9m24.497501705s) [noise=true , address=104.36.82.55:4194 , piece_id=FNJCFJJ3HRW2DLPM7EN3JP3H5ULOML35UINCBURYSPCMTJPUO4VA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.774977909 (60h9m24.497540939s) [noise=true , address=104.36.82.52:4099 , piece_id=M36SDTUWEJ56462XRHIDZ2KT4HNEGUQJMBRNPBF23AM5GENU3XBQ, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: rpc: dial tcp 104.36.82.52:4099: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775013026 (60h9m24.497576066s) [noise=true , address=104.36.82.46:4506 , piece_id=6A3NVN5EP6LXPYNIY3AS26YG4MQCG5XFSS7SESLNJNJCWRIS75DA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775045699 (60h9m24.497608739s) [noise=true , address=77.90.31.16:4766 , piece_id=CZRX5R553LQRF337VHGAXVOQQ6QZX54DLCXI75R44MZKGBEQ4LYA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 188.166.225.170:34380->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775071418 (60h9m24.497634458s) [noise=true , address=104.36.82.54:4176 , piece_id=NWBOBIRRB7BTGD42SJLJXE5MPQSEQGDCVTQV2GOBW4MWORSJNBIA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775098740 (60h9m24.49766178s) [noise=true , address=104.36.82.56:4263 , piece_id=KP7NCM3YPEODH7FQPZULNAHLY2GEAMYWTMHCSTA5SQCHOKLNZGDA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775106897 (60h9m24.497669937s) [noise=true , address=77.90.31.10:4528 , piece_id=U5ZBTFMYANJ34HWP2LQZSITDT6A4ZTG34LFU3L4ERMS6SFIA6PAA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: rpc: dial tcp 77.90.31.10:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775112166 (60h9m24.497675206s) [noise=true , address=104.36.82.13:4194 , piece_id=HIOCKLJCZTSHZEM27XUNCXFBXE2G6AROTOAXTZAHDIRRFJJ6MG6A, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775119430 (60h9m24.49768246s) [noise=true , address=104.36.82.47:4541 , piece_id=ZYY3HJO3M5WIOMIIMTSDMGVQ3QNWUH52V7XHZLYHMBS7DIJBXWQA, seg_pos={0 0}, upload=57697]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:14.775131874 (60h9m24.497694914s) [seg_pos={0 0}, upload=57697]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:45:15.010003906 (60h9m24.732566946s) [upload=57697]: Done uploading segments... Jul 26 23:45:15.010027131 (60h9m24.732590171s) [upload=57697]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236232516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..............XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....] duration: 8.640580073s FAIL (2025-07-26 23:45:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.640593809s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [.X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 6.055279808s FAIL (2025-07-26 23:45:06 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 9302307 (60h7m51.291865367s) [noise=true , address=104.36.82.8:4972 , piece_id=ZLDAUEK4NYG6RAE4KEULX7WIDUHPG7HLNEKFLBFIEFTILFDEL6KQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569316373 (60h7m51.291879413s) [seg_pos={0 0}, upload=57680]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:43:41.805102594 (60h7m51.527665634s) [upload=57680]: Done uploading segments... Jul 26 23:43:41.805130116 (60h7m51.527693156s) [upload=57680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:44:10.982201867 (60h8m20.704764908s) [upload=57696]: Starting upload Jul 26 23:44:10.982313722 (60h8m20.704876762s) [upload=57696]: Done starting upload Jul 26 23:44:10.982443570 (60h8m20.70500665s) [upload=57696]: Uploading segments... Jul 26 23:44:10.982688348 (60h8m20.705251388s) [upload=57696]: Got next segment. Inline: false Jul 26 23:44:10.982727103 (60h8m20.705290153s) [upload=57696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:44:10.982734507 (60h8m20.705297547s) [upload=57696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:44:11.213907972 (60h8m20.936471012s) [seg_pos={0 0}, upload=57696]: Begin upload segment... Jul 26 23:44:11.213953648 (60h8m20.936516698s) [seg_pos={0 0}, upload=57696]: Done begin upload segment. Jul 26 23:44:11.213987413 (60h8m20.936550453s) [noise=true , address=45.91.116.13:4155 , piece_id=DUK27LC53F4ZR2N75MMUN4YCDIHQ4OBFTX5K5RG4P4WKIBBIH7KA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214010828 (60h8m20.936573858s) [noise=true , address=45.91.116.14:4191 , piece_id=DU37DAAQEFDZR2HR65QW4WLZ52GOLS44DPWMSKRWLE5B5VJWYUFA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214383852 (60h8m20.936946892s) [noise=true , address=45.91.116.5:4298 , piece_id=X2JHCXB2AQ6S55O2YQF5YFEFO7FCB5EPRY26CS6EJKK56PQDMYDA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214439508 (60h8m20.937002548s) [noise=true , address=104.36.82.11:4113 , piece_id=EXFVK4XJPUEO5P3YGZAWKEXFS6TP6LIYBCRNFRHZKU3HNFJQBBYA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214641645 (60h8m20.937204685s) [noise=true , address=104.36.82.16:4293 , piece_id=NLYCBP5LYFYCG4ZI2L2D4R3BTXDUAO3LG3V5ESWSMXIB6CAV5X2Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214737499 (60h8m20.937300539s) [noise=true , address=77.90.31.14:4662 , piece_id=CUCD3MESH5SQN476D64JL74BZGAGOQ2PEOGJBFBEQU5HKRCU4QKA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214959072 (60h8m20.937522112s) [noise=true , address=77.90.31.11:4553 , piece_id=OTDGZ5BEAJKRRCVCB4WTLQY7XMJEQLFSLJ4KJIQQE27YCOFJRM4Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215013817 (60h8m20.937576847s) [noise=true , address=45.91.116.13:4154 , piece_id=YLZUD6V7BOEQNJQFDSB5JENHJW7MDTP5242MVBIOSPT7WPU66OTQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215222857 (60h8m20.937785897s) [noise=true , address=77.90.31.15:4698 , piece_id=EGXALWCXXQTDEEE4XEFINZSMXEU3PHQF27HQJQXMPZHZSPQ7OIRQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215468266 (60h8m20.938031296s) [noise=true , address=45.91.116.8:4048 , piece_id=OWXB4SRGD6FD5OGPVRCNAARYA7ZCS3O54TQVOCIKTMYE4SCJ4EYA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215539122 (60h8m20.938102162s) [noise=true , address=45.91.116.10:4083 , piece_id=M5RHJRNEOZL4FYDKGXT5AHPV3H74VJ7RDLQPFC3MA3ILYDNUIEVQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215669742 (60h8m20.938232782s) [noise=true , address=77.90.31.19:4843 , piece_id=CJ4LLYVV3RL4PX4DNDBBXA2MYBNMZBKOUEW2VABOHEMMLXTODGYQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215876768 (60h8m20.938439808s) [noise=true , address=45.91.116.9:4262 , piece_id=AKY5RJHHOCYSW6B6BGHTTKYOHFNOIGR4ED2OJO5U3Y3GB7OJ342Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216071300 (60h8m20.93863434s) [noise=true , address=77.90.31.10:4517 , piece_id=UPKBHQG6WDKJ2WHAUBE7DR5BUJGEW7VCCU75ZB3ZM5WELJ3JBUAQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216212781 (60h8m20.938775861s) [noise=true , address=45.91.116.14:4190 , piece_id=KSAUHIHHL5NWM4XWA36VMLXUYG3FBVB7ZAXHHT3MSQBITFLPBKBA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216765318 (60h8m20.939328378s) [noise=true , address=45.91.116.4:4937 , piece_id=AEHDGRLANUG7HBZ6DJQAFCFRRVHFKSDOOCIXQ6HASSN67AIJGL6Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:12.921025138 (60h8m22.643588178s) [upload=57696]: Next returned nil segment Jul 26 23:44:12.921046179 (60h8m22.643609219s) [upload=57696]: Waiting for error group managing segments... Jul 26 23:44:13.252975038 (60h8m22.975538078s) [noise=true , address=45.91.116.4:4937 , piece_id=AEHDGRLANUG7HBZ6DJQAFCFRRVHFKSDOOCIXQ6HASSN67AIJGL6Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.312252274 (60h8m23.034815324s) [noise=true , address=45.91.116.8:4048 , piece_id=OWXB4SRGD6FD5OGPVRCNAARYA7ZCS3O54TQVOCIKTMYE4SCJ4EYA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.319210898 (60h8m23.041773938s) [noise=true , address=45.91.116.13:4154 , piece_id=YLZUD6V7BOEQNJQFDSB5JENHJW7MDTP5242MVBIOSPT7WPU66OTQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.323519492 (60h8m23.046082532s) [noise=true , address=77.90.31.11:4553 , piece_id=OTDGZ5BEAJKRRCVCB4WTLQY7XMJEQLFSLJ4KJIQQE27YCOFJRM4Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.374341870 (60h8m23.09690491s) [noise=true , address=45.91.116.9:4262 , piece_id=AKY5RJHHOCYSW6B6BGHTTKYOHFNOIGR4ED2OJO5U3Y3GB7OJ342Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.374945334 (60h8m23.097508374s) [noise=true , address=45.91.116.5:4298 , piece_id=X2JHCXB2AQ6S55O2YQF5YFEFO7FCB5EPRY26CS6EJKK56PQDMYDA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.384038252 (60h8m23.106601292s) [noise=true , address=77.90.31.14:4662 , piece_id=CUCD3MESH5SQN476D64JL74BZGAGOQ2PEOGJBFBEQU5HKRCU4QKA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.384810109 (60h8m23.107373149s) [noise=true , address=45.91.116.10:4083 , piece_id=M5RHJRNEOZL4FYDKGXT5AHPV3H74VJ7RDLQPFC3MA3ILYDNUIEVQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.419836365 (60h8m23.142399405s) [noise=true , address=45.91.116.13:4155 , piece_id=DUK27LC53F4ZR2N75MMUN4YCDIHQ4OBFTX5K5RG4P4WKIBBIH7KA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.432089228 (60h8m23.154652269s) [noise=true , address=45.91.116.14:4190 , piece_id=KSAUHIHHL5NWM4XWA36VMLXUYG3FBVB7ZAXHHT3MSQBITFLPBKBA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.470675345 (60h8m23.193238385s) [noise=true , address=45.91.116.14:4191 , piece_id=DU37DAAQEFDZR2HR65QW4WLZ52GOLS44DPWMSKRWLE5B5VJWYUFA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:15.469449770 (60h8m25.19201281s) [noise=true , address=77.90.31.19:4843 , piece_id=CJ4LLYVV3RL4PX4DNDBBXA2MYBNMZBKOUEW2VABOHEMMLXTODGYQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:38.748913596 (60h8m48.471476636s) [noise=true , address=77.90.31.15:4698 , piece_id=EGXALWCXXQTDEEE4XEFINZSMXEU3PHQF27HQJQXMPZHZSPQ7OIRQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:38.748947441 (60h8m48.471510491s) [seg_pos={0 0}, upload=57696]: Segment reached optimal threshold of 13 pieces. Jul 26 23:44:38.749438781 (60h8m48.472001861s) [noise=true , address=104.36.82.16:4293 , piece_id=NLYCBP5LYFYCG4ZI2L2D4R3BTXDUAO3LG3V5ESWSMXIB6CAV5X2Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:57172->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749495509 (60h8m48.472058569s) [noise=true , address=104.36.82.11:4113 , piece_id=EXFVK4XJPUEO5P3YGZAWKEXFS6TP6LIYBCRNFRHZKU3HNFJQBBYA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: rpc: dial tcp 104.36.82.11:4113: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749550935 (60h8m48.472113985s) [noise=true , address=77.90.31.10:4517 , piece_id=UPKBHQG6WDKJ2WHAUBE7DR5BUJGEW7VCCU75ZB3ZM5WELJ3JBUAQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:56940->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749579991 (60h8m48.472143051s) [seg_pos={0 0}, upload=57696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:44:38.749667959 (60h8m48.472231039s) [upload=57696]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:44:38.749675774 (60h8m48.472238844s) [upload=57696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:44:39.025096356 (60h8m48.747659396s) [upload=57696]: Done uploading segments... Jul 26 23:44:39.025111986 (60h8m48.747675026s) [upload=57696]: Upload finished. err: <nil> Duration: 6.05530694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.....XX..XX.XX..XXX............XXXX.....XXX..X.....X......X.] duration: 29.13797234s FAIL (2025-07-26 23:44:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.138024519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.832006919s FAIL (2025-07-26 23:44:46 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.832027809s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [..X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001786558s FAIL (2025-07-26 23:44:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 9302307 (60h7m51.291865367s) [noise=true , address=104.36.82.8:4972 , piece_id=ZLDAUEK4NYG6RAE4KEULX7WIDUHPG7HLNEKFLBFIEFTILFDEL6KQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569316373 (60h7m51.291879413s) [seg_pos={0 0}, upload=57680]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:43:41.805102594 (60h7m51.527665634s) [upload=57680]: Done uploading segments... Jul 26 23:43:41.805130116 (60h7m51.527693156s) [upload=57680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:44:10.982201867 (60h8m20.704764908s) [upload=57696]: Starting upload Jul 26 23:44:10.982313722 (60h8m20.704876762s) [upload=57696]: Done starting upload Jul 26 23:44:10.982443570 (60h8m20.70500665s) [upload=57696]: Uploading segments... Jul 26 23:44:10.982688348 (60h8m20.705251388s) [upload=57696]: Got next segment. Inline: false Jul 26 23:44:10.982727103 (60h8m20.705290153s) [upload=57696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:44:10.982734507 (60h8m20.705297547s) [upload=57696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:44:11.213907972 (60h8m20.936471012s) [seg_pos={0 0}, upload=57696]: Begin upload segment... Jul 26 23:44:11.213953648 (60h8m20.936516698s) [seg_pos={0 0}, upload=57696]: Done begin upload segment. Jul 26 23:44:11.213987413 (60h8m20.936550453s) [noise=true , address=45.91.116.13:4155 , piece_id=DUK27LC53F4ZR2N75MMUN4YCDIHQ4OBFTX5K5RG4P4WKIBBIH7KA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214010828 (60h8m20.936573858s) [noise=true , address=45.91.116.14:4191 , piece_id=DU37DAAQEFDZR2HR65QW4WLZ52GOLS44DPWMSKRWLE5B5VJWYUFA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214383852 (60h8m20.936946892s) [noise=true , address=45.91.116.5:4298 , piece_id=X2JHCXB2AQ6S55O2YQF5YFEFO7FCB5EPRY26CS6EJKK56PQDMYDA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214439508 (60h8m20.937002548s) [noise=true , address=104.36.82.11:4113 , piece_id=EXFVK4XJPUEO5P3YGZAWKEXFS6TP6LIYBCRNFRHZKU3HNFJQBBYA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214641645 (60h8m20.937204685s) [noise=true , address=104.36.82.16:4293 , piece_id=NLYCBP5LYFYCG4ZI2L2D4R3BTXDUAO3LG3V5ESWSMXIB6CAV5X2Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214737499 (60h8m20.937300539s) [noise=true , address=77.90.31.14:4662 , piece_id=CUCD3MESH5SQN476D64JL74BZGAGOQ2PEOGJBFBEQU5HKRCU4QKA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.214959072 (60h8m20.937522112s) [noise=true , address=77.90.31.11:4553 , piece_id=OTDGZ5BEAJKRRCVCB4WTLQY7XMJEQLFSLJ4KJIQQE27YCOFJRM4Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215013817 (60h8m20.937576847s) [noise=true , address=45.91.116.13:4154 , piece_id=YLZUD6V7BOEQNJQFDSB5JENHJW7MDTP5242MVBIOSPT7WPU66OTQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215222857 (60h8m20.937785897s) [noise=true , address=77.90.31.15:4698 , piece_id=EGXALWCXXQTDEEE4XEFINZSMXEU3PHQF27HQJQXMPZHZSPQ7OIRQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215468266 (60h8m20.938031296s) [noise=true , address=45.91.116.8:4048 , piece_id=OWXB4SRGD6FD5OGPVRCNAARYA7ZCS3O54TQVOCIKTMYE4SCJ4EYA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215539122 (60h8m20.938102162s) [noise=true , address=45.91.116.10:4083 , piece_id=M5RHJRNEOZL4FYDKGXT5AHPV3H74VJ7RDLQPFC3MA3ILYDNUIEVQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215669742 (60h8m20.938232782s) [noise=true , address=77.90.31.19:4843 , piece_id=CJ4LLYVV3RL4PX4DNDBBXA2MYBNMZBKOUEW2VABOHEMMLXTODGYQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.215876768 (60h8m20.938439808s) [noise=true , address=45.91.116.9:4262 , piece_id=AKY5RJHHOCYSW6B6BGHTTKYOHFNOIGR4ED2OJO5U3Y3GB7OJ342Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216071300 (60h8m20.93863434s) [noise=true , address=77.90.31.10:4517 , piece_id=UPKBHQG6WDKJ2WHAUBE7DR5BUJGEW7VCCU75ZB3ZM5WELJ3JBUAQ, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216212781 (60h8m20.938775861s) [noise=true , address=45.91.116.14:4190 , piece_id=KSAUHIHHL5NWM4XWA36VMLXUYG3FBVB7ZAXHHT3MSQBITFLPBKBA, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:11.216765318 (60h8m20.939328378s) [noise=true , address=45.91.116.4:4937 , piece_id=AEHDGRLANUG7HBZ6DJQAFCFRRVHFKSDOOCIXQ6HASSN67AIJGL6Q, seg_pos={0 0}, upload=57696]: Uploading piece... Jul 26 23:44:12.921025138 (60h8m22.643588178s) [upload=57696]: Next returned nil segment Jul 26 23:44:12.921046179 (60h8m22.643609219s) [upload=57696]: Waiting for error group managing segments... Jul 26 23:44:13.252975038 (60h8m22.975538078s) [noise=true , address=45.91.116.4:4937 , piece_id=AEHDGRLANUG7HBZ6DJQAFCFRRVHFKSDOOCIXQ6HASSN67AIJGL6Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.312252274 (60h8m23.034815324s) [noise=true , address=45.91.116.8:4048 , piece_id=OWXB4SRGD6FD5OGPVRCNAARYA7ZCS3O54TQVOCIKTMYE4SCJ4EYA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.319210898 (60h8m23.041773938s) [noise=true , address=45.91.116.13:4154 , piece_id=YLZUD6V7BOEQNJQFDSB5JENHJW7MDTP5242MVBIOSPT7WPU66OTQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.323519492 (60h8m23.046082532s) [noise=true , address=77.90.31.11:4553 , piece_id=OTDGZ5BEAJKRRCVCB4WTLQY7XMJEQLFSLJ4KJIQQE27YCOFJRM4Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.374341870 (60h8m23.09690491s) [noise=true , address=45.91.116.9:4262 , piece_id=AKY5RJHHOCYSW6B6BGHTTKYOHFNOIGR4ED2OJO5U3Y3GB7OJ342Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.374945334 (60h8m23.097508374s) [noise=true , address=45.91.116.5:4298 , piece_id=X2JHCXB2AQ6S55O2YQF5YFEFO7FCB5EPRY26CS6EJKK56PQDMYDA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.384038252 (60h8m23.106601292s) [noise=true , address=77.90.31.14:4662 , piece_id=CUCD3MESH5SQN476D64JL74BZGAGOQ2PEOGJBFBEQU5HKRCU4QKA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.384810109 (60h8m23.107373149s) [noise=true , address=45.91.116.10:4083 , piece_id=M5RHJRNEOZL4FYDKGXT5AHPV3H74VJ7RDLQPFC3MA3ILYDNUIEVQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.419836365 (60h8m23.142399405s) [noise=true , address=45.91.116.13:4155 , piece_id=DUK27LC53F4ZR2N75MMUN4YCDIHQ4OBFTX5K5RG4P4WKIBBIH7KA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.432089228 (60h8m23.154652269s) [noise=true , address=45.91.116.14:4190 , piece_id=KSAUHIHHL5NWM4XWA36VMLXUYG3FBVB7ZAXHHT3MSQBITFLPBKBA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:13.470675345 (60h8m23.193238385s) [noise=true , address=45.91.116.14:4191 , piece_id=DU37DAAQEFDZR2HR65QW4WLZ52GOLS44DPWMSKRWLE5B5VJWYUFA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:15.469449770 (60h8m25.19201281s) [noise=true , address=77.90.31.19:4843 , piece_id=CJ4LLYVV3RL4PX4DNDBBXA2MYBNMZBKOUEW2VABOHEMMLXTODGYQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:38.748913596 (60h8m48.471476636s) [noise=true , address=77.90.31.15:4698 , piece_id=EGXALWCXXQTDEEE4XEFINZSMXEU3PHQF27HQJQXMPZHZSPQ7OIRQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: <nil> Jul 26 23:44:38.748947441 (60h8m48.471510491s) [seg_pos={0 0}, upload=57696]: Segment reached optimal threshold of 13 pieces. Jul 26 23:44:38.749438781 (60h8m48.472001861s) [noise=true , address=104.36.82.16:4293 , piece_id=NLYCBP5LYFYCG4ZI2L2D4R3BTXDUAO3LG3V5ESWSMXIB6CAV5X2Q, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:57172->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749495509 (60h8m48.472058569s) [noise=true , address=104.36.82.11:4113 , piece_id=EXFVK4XJPUEO5P3YGZAWKEXFS6TP6LIYBCRNFRHZKU3HNFJQBBYA, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: rpc: dial tcp 104.36.82.11:4113: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749550935 (60h8m48.472113985s) [noise=true , address=77.90.31.10:4517 , piece_id=UPKBHQG6WDKJ2WHAUBE7DR5BUJGEW7VCCU75ZB3ZM5WELJ3JBUAQ, seg_pos={0 0}, upload=57696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:56940->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:38.749579991 (60h8m48.472143051s) [seg_pos={0 0}, upload=57696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:44:38.749667959 (60h8m48.472231039s) [upload=57696]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:44:38.749675774 (60h8m48.472238844s) [upload=57696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:44:39.025096356 (60h8m48.747659396s) [upload=57696]: Done uploading segments... Jul 26 23:44:39.025111986 (60h8m48.747675026s) [upload=57696]: Upload finished. err: <nil> Duration: 30.001805263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237334229s FAIL (2025-07-26 23:44:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.237361352s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236012712s FAIL (2025-07-26 23:44:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 733656s) [seg_pos={0 0}, upload=57681]: Begin upload segment... Jul 26 23:43:15.414215081 (60h7m25.136778111s) [seg_pos={0 0}, upload=57681]: Done begin upload segment. Jul 26 23:43:15.414227485 (60h7m25.136790525s) [noise=true , address=104.36.82.54:4158 , piece_id=N4ZB6NMKFK3WQDIJNDPGGLC2LHUVJS63TXSFUXX7P7Z2MD6ATNTQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.414238857 (60h7m25.136801897s) [noise=true , address=79.127.220.101:4041 , piece_id=LLS2ZA2OPUBGBUKE3K76HZP7WEIKUBDHOWEFWQFYZOYZ63FV66KQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.414581091 (60h7m25.137144131s) [noise=true , address=79.127.205.235:4686 , piece_id=FDDULO3CTYICJJGID7XMDC6MGJG7HA5NRSZSCV3V7XWNUGJHSVXQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.414595940 (60h7m25.13715898s) [noise=true , address=79.127.201.214:4289 , piece_id=YLFG642ZNFCABWAFBTUVKDHP7KHVKQT2OZKPHWUVC53DWSHFXUJQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.414841339 (60h7m25.137404379s) [noise=true , address=104.36.82.56:4234 , piece_id=RVROPCQ73GBGS3FG6BWPVZOFICB7CFTZDGBGYPVUI44ISOJDL5GQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.415094304 (60h7m25.137657334s) [noise=true , address=64.49.13.140:4508 , piece_id=WVE3K6L5CZCXRW57544CESTJW6ZY6NW3LWJFIH4ROWESUKNQWSKA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.415354371 (60h7m25.137917411s) [noise=true , address=77.90.31.18:4817 , piece_id=ZIFIQFRHJVGFVOVZ72HSSDXPUTR72UZPYKCLD3P7KYVFL75HA6YQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.415601444 (60h7m25.138164484s) [noise=true , address=77.90.31.10:4538 , piece_id=PAM5WK3SS2OQICTZHYPYJPA552GYNUFA37KRHJ2HYNMXSC6IJRKQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.415658092 (60h7m25.138221132s) [noise=true , address=104.36.82.46:4490 , piece_id=7BNAA4VGKB6B22PQEYUZQ6ODSBBI3PZ35U4WKUI5U4FFQXCGAN6Q, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.415970791 (60h7m25.138533831s) [noise=true , address=77.90.31.21:4904 , piece_id=T5FOGPWE6D65ARFP4YWNV3AGI6IEJX5ZTJ7EOO6ZUSQZG4V2E4IA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.416443505 (60h7m25.139006545s) [noise=true , address=79.127.205.245:4238 , piece_id=DOXX4VWX76BNA3BYKP4C4IRP7WOMTWW3HYVCRJMMQAXDLRKH5PTQ, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.416476147 (60h7m25.139039187s) [noise=true , address=79.127.205.234:4612 , piece_id=FKOGGYMRTX2VE53ROJZRQZXFITAQNUH7YR23JP4C2DJODXI6NMIA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.416696108 (60h7m25.139259148s) [noise=true , address=79.127.205.246:4275 , piece_id=Y6T5M7BQXXFG3VVI24XUGUMCGP55JZOQMIK6H3GC22KJMGCRVUZA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.416885921 (60h7m25.139448961s) [noise=true , address=77.90.31.12:4621 , piece_id=RZXOD3IRDJQE272VDEYUVKF23SVPKG6JQLKQVA4ZMIJDEWJCCKYA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.417084461 (60h7m25.139647491s) [noise=true , address=104.36.82.47:4521 , piece_id=J6CKSNDLS6X44MRMV2PQSCI3LUBONWDCH2MWBTI3443W3X3M3LYA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.417114488 (60h7m25.139677518s) [noise=true , address=104.36.82.14:4236 , piece_id=QRKTPRKS3NRA4WXNLTNXJY2SVYQNAGPITYMD5S45SYDMGUSKKNXA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.417608554 (60h7m25.140171594s) [noise=true , address=79.127.219.38:4015 , piece_id=5SSMM5XCJS5G7O27Z6VCQXXG7X5B22YCH3INCSFJZSXAMRLWULIA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:15.417823796 (60h7m25.140386836s) [noise=true , address=79.127.201.215:4356 , piece_id=NPINC7YPONFRJUHW3EV37NUQBSOZ6OMS2IEDKFRLO7UG7P6IGJFA, seg_pos={0 0}, upload=57681]: Uploading piece... Jul 26 23:43:16.432432157 (60h7m26.154995197s) [upload=57681]: Next returned nil segment Jul 26 23:43:16.432445342 (60h7m26.155008382s) [upload=57681]: Waiting for error group managing segments... Jul 26 23:43:16.801024885 (60h7m26.523587965s) [noise=true , address=79.127.205.246:4275 , piece_id=Y6T5M7BQXXFG3VVI24XUGUMCGP55JZOQMIK6H3GC22KJMGCRVUZA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:16.816668977 (60h7m26.539232017s) [noise=true , address=79.127.205.234:4612 , piece_id=FKOGGYMRTX2VE53ROJZRQZXFITAQNUH7YR23JP4C2DJODXI6NMIA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:16.829526507 (60h7m26.552089537s) [noise=true , address=79.127.205.235:4686 , piece_id=FDDULO3CTYICJJGID7XMDC6MGJG7HA5NRSZSCV3V7XWNUGJHSVXQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:16.911117806 (60h7m26.633680846s) [noise=true , address=79.127.205.245:4238 , piece_id=DOXX4VWX76BNA3BYKP4C4IRP7WOMTWW3HYVCRJMMQAXDLRKH5PTQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:17.491559155 (60h7m27.214122195s) [noise=true , address=79.127.219.38:4015 , piece_id=5SSMM5XCJS5G7O27Z6VCQXXG7X5B22YCH3INCSFJZSXAMRLWULIA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:17.533852364 (60h7m27.256415404s) [noise=true , address=77.90.31.12:4621 , piece_id=RZXOD3IRDJQE272VDEYUVKF23SVPKG6JQLKQVA4ZMIJDEWJCCKYA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:17.563379258 (60h7m27.285942298s) [noise=true , address=77.90.31.18:4817 , piece_id=ZIFIQFRHJVGFVOVZ72HSSDXPUTR72UZPYKCLD3P7KYVFL75HA6YQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:19.044133014 (60h7m28.766696044s) [noise=true , address=79.127.220.101:4041 , piece_id=LLS2ZA2OPUBGBUKE3K76HZP7WEIKUBDHOWEFWQFYZOYZ63FV66KQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:19.290050188 (60h7m29.012613238s) [noise=true , address=64.49.13.140:4508 , piece_id=WVE3K6L5CZCXRW57544CESTJW6ZY6NW3LWJFIH4ROWESUKNQWSKA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:20.113871755 (60h7m29.836434785s) [noise=true , address=79.127.201.214:4289 , piece_id=YLFG642ZNFCABWAFBTUVKDHP7KHVKQT2OZKPHWUVC53DWSHFXUJQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:43:50.151202992 (60h7m59.873766072s) [noise=true , address=79.127.201.215:4356 , piece_id=NPINC7YPONFRJUHW3EV37NUQBSOZ6OMS2IEDKFRLO7UG7P6IGJFA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:44:05.299024031 (60h8m15.021587071s) [noise=true , address=77.90.31.21:4904 , piece_id=T5FOGPWE6D65ARFP4YWNV3AGI6IEJX5ZTJ7EOO6ZUSQZG4V2E4IA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: <nil> Jul 26 23:44:14.319770510 (60h8m24.04233355s) [noise=true , address=104.36.82.46:4490 , piece_id=7BNAA4VGKB6B22PQEYUZQ6ODSBBI3PZ35U4WKUI5U4FFQXCGAN6Q, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 188.166.225.170:43198->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319811549 (60h8m24.042374589s) [noise=true , address=104.36.82.54:4158 , piece_id=N4ZB6NMKFK3WQDIJNDPGGLC2LHUVJS63TXSFUXX7P7Z2MD6ATNTQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 188.166.225.170:50150->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319840745 (60h8m24.042403775s) [noise=true , address=104.36.82.14:4236 , piece_id=QRKTPRKS3NRA4WXNLTNXJY2SVYQNAGPITYMD5S45SYDMGUSKKNXA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:1XaE29W27ym2RmuGdBvHeKcQZZEUbask1zsm9Bz8xLokZNGfcj, address:104.36.82.14:4236): protocol: write tcp 188.166.225.170:40424->104.36.82.14:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319880431 (60h8m24.042443461s) [noise=true , address=77.90.31.10:4538 , piece_id=PAM5WK3SS2OQICTZHYPYJPA552GYNUFA37KRHJ2HYNMXSC6IJRKQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: write tcp 188.166.225.170:45526->77.90.31.10:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319905358 (60h8m24.042468388s) [noise=true , address=104.36.82.47:4521 , piece_id=J6CKSNDLS6X44MRMV2PQSCI3LUBONWDCH2MWBTI3443W3X3M3LYA, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: rpc: dial tcp 104.36.82.47:4521: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319925357 (60h8m24.042488397s) [noise=true , address=104.36.82.56:4234 , piece_id=RVROPCQ73GBGS3FG6BWPVZOFICB7CFTZDGBGYPVUI44ISOJDL5GQ, seg_pos={0 0}, upload=57681]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:14.319956116 (60h8m24.042519156s) [seg_pos={0 0}, upload=57681]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:44:14.554603274 (60h8m24.277166314s) [upload=57681]: Done uploading segments... Jul 26 23:44:14.554626789 (60h8m24.277189829s) [upload=57681]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.236023963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [......XX..XX.XX..XXX............XXXX.....XXX..X.....X......X] duration: 1m0.000743176s FAIL (2025-07-26 23:44:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000756622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...............XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X...] duration: 7.491901947s FAIL (2025-07-26 23:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.491923619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.036461082s FAIL (2025-07-26 23:43:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.036482392s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [...X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.238328561s FAIL (2025-07-26 23:43:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== l 26 23:43:12.214995033 (60h7m21.937558073s) [upload=57680]: Uploading segments... Jul 26 23:43:12.215124541 (60h7m21.937687571s) [upload=57680]: Got next segment. Inline: false Jul 26 23:43:12.215148016 (60h7m21.937711056s) [upload=57680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:12.215152475 (60h7m21.937715505s) [upload=57680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:12.449243251 (60h7m22.171806291s) [seg_pos={0 0}, upload=57680]: Begin upload segment... Jul 26 23:43:12.449283457 (60h7m22.171846497s) [seg_pos={0 0}, upload=57680]: Done begin upload segment. Jul 26 23:43:12.449303446 (60h7m22.171866486s) [noise=true , address=104.36.82.8:4972 , piece_id=ZLDAUEK4NYG6RAE4KEULX7WIDUHPG7HLNEKFLBFIEFTILFDEL6KQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.449311942 (60h7m22.171874982s) [noise=true , address=45.91.116.10:4082 , piece_id=4Y7Y6EPV53XQXWEG2MIVOC7LXMXQ352GYOV44WEYSAJ3ZGDMYYSQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.449668434 (60h7m22.172231474s) [noise=true , address=77.90.31.14:4661 , piece_id=T72M4GYTK2V6FFEO6KIEJVKXGE5MKAFUG6XQPW2F64NSO6WBBSNA, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.449689975 (60h7m22.172253005s) [noise=true , address=104.36.82.2:4732 , piece_id=GLXFEBT6JZXST46XBWXTDY6LUD6Q44RUFOG2KOTZPR3NG5SHW4QQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.449945534 (60h7m22.172508574s) [noise=true , address=104.36.82.14:4222 , piece_id=FASS26AOG4NIFZJAN7TUGOYMIEVZBWZ7S4OY5MRNJERBRD4RJ66Q, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.449981954 (60h7m22.172544994s) [noise=true , address=45.91.116.6:4975 , piece_id=L64QESHNTIHCX2OVPC3HLMV26D3KAYUH4IHDKDLNZH3R3AQUM52Q, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450194901 (60h7m22.172757941s) [noise=true , address=45.91.116.5:4297 , piece_id=ZNITBZB7AWJ6MUDFK34FWECPCBL4H4MBYHY2TEWQNKGYYF75U6FQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450292027 (60h7m22.172855067s) [noise=true , address=104.36.82.5:4838 , piece_id=NE5BOA52B7C7ZZC3BMRMHDNAYJPXPGOEBO7VNMYS25ZOTYDQQV2Q, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450434229 (60h7m22.172997269s) [noise=true , address=104.36.82.4:4801 , piece_id=P5WZ6SEFXUE5RQOASDEQOQXMBAZ63SNHMNO2LS4YRLPBSW2BDWKA, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450567945 (60h7m22.173130985s) [noise=true , address=104.36.82.3:4765 , piece_id=YPLBSGCN6EGBB2IW32RLZBZKGHHNUJPVT6JJUOQ7TMRD475TAYGA, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450661865 (60h7m22.173224905s) [noise=true , address=104.36.82.9:4005 , piece_id=ZEDBJG4CSWZYCC25WKJWPXH3K2G66NCBETE2OR6KIASKKOSYEPOA, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450839355 (60h7m22.173402395s) [noise=true , address=104.36.82.7:4910 , piece_id=TUWPIBUI7SCRLTOZAP6OKHRWITGNBYHZUDAFNISCP7UF5IN7ACVQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.450891575 (60h7m22.173454615s) [noise=true , address=45.91.116.12:4225 , piece_id=GMRLZP76DHDIZCF2I25QHYENLG5YM7SUDEDHL3OMZR3BC7LV63IA, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.451105514 (60h7m22.173668554s) [noise=true , address=45.91.116.14:4191 , piece_id=O3AA65QDWVPDCA2YPDNSZ6DC5HFFI4RCM2OUPP4FF7BVAYW6IGDQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.451115763 (60h7m22.173678804s) [noise=true , address=104.36.82.5:4837 , piece_id=L7XOW4TLACVBYVBB6POFHV7QNMIVR5D3URS2YY35Z7RMYQZPHO7Q, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:12.451328320 (60h7m22.17389136s) [noise=true , address=45.91.116.4:4940 , piece_id=6ASWKLZOB5TRVL66DYGKHLRUVTMNSXTROCTB3HM5NM2ZZDKDL2ZQ, seg_pos={0 0}, upload=57680]: Uploading piece... Jul 26 23:43:14.179568542 (60h7m23.902131582s) [upload=57680]: Next returned nil segment Jul 26 23:43:14.179589572 (60h7m23.902152612s) [upload=57680]: Waiting for error group managing segments... Jul 26 23:43:14.490073149 (60h7m24.21263619s) [noise=true , address=45.91.116.6:4975 , piece_id=L64QESHNTIHCX2OVPC3HLMV26D3KAYUH4IHDKDLNZH3R3AQUM52Q, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.519191202 (60h7m24.241754242s) [noise=true , address=45.91.116.5:4297 , piece_id=ZNITBZB7AWJ6MUDFK34FWECPCBL4H4MBYHY2TEWQNKGYYF75U6FQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.543930246 (60h7m24.266493286s) [noise=true , address=45.91.116.12:4225 , piece_id=GMRLZP76DHDIZCF2I25QHYENLG5YM7SUDEDHL3OMZR3BC7LV63IA, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.545996769 (60h7m24.268559809s) [noise=true , address=45.91.116.10:4082 , piece_id=4Y7Y6EPV53XQXWEG2MIVOC7LXMXQ352GYOV44WEYSAJ3ZGDMYYSQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.546228993 (60h7m24.268792023s) [noise=true , address=45.91.116.14:4191 , piece_id=O3AA65QDWVPDCA2YPDNSZ6DC5HFFI4RCM2OUPP4FF7BVAYW6IGDQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.555781059 (60h7m24.278344099s) [noise=true , address=45.91.116.4:4940 , piece_id=6ASWKLZOB5TRVL66DYGKHLRUVTMNSXTROCTB3HM5NM2ZZDKDL2ZQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:14.559463003 (60h7m24.282026044s) [noise=true , address=77.90.31.14:4661 , piece_id=T72M4GYTK2V6FFEO6KIEJVKXGE5MKAFUG6XQPW2F64NSO6WBBSNA, seg_pos={0 0}, upload=57680]: Done uploading piece. err: <nil> Jul 26 23:43:41.569136700 (60h7m51.29169974s) [noise=true , address=104.36.82.14:4222 , piece_id=FASS26AOG4NIFZJAN7TUGOYMIEVZBWZ7S4OY5MRNJERBRD4RJ66Q, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:47924->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569180072 (60h7m51.291743102s) [noise=true , address=104.36.82.4:4801 , piece_id=P5WZ6SEFXUE5RQOASDEQOQXMBAZ63SNHMNO2LS4YRLPBSW2BDWKA, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569209709 (60h7m51.291772739s) [noise=true , address=104.36.82.9:4005 , piece_id=ZEDBJG4CSWZYCC25WKJWPXH3K2G66NCBETE2OR6KIASKKOSYEPOA, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569226021 (60h7m51.291789051s) [noise=true , address=104.36.82.3:4765 , piece_id=YPLBSGCN6EGBB2IW32RLZBZKGHHNUJPVT6JJUOQ7TMRD475TAYGA, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569232262 (60h7m51.291795302s) [noise=true , address=104.36.82.2:4732 , piece_id=GLXFEBT6JZXST46XBWXTDY6LUD6Q44RUFOG2KOTZPR3NG5SHW4QQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:48998->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569261969 (60h7m51.291825009s) [noise=true , address=104.36.82.5:4838 , piece_id=NE5BOA52B7C7ZZC3BMRMHDNAYJPXPGOEBO7VNMYS25ZOTYDQQV2Q, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569266768 (60h7m51.291829808s) [noise=true , address=104.36.82.7:4910 , piece_id=TUWPIBUI7SCRLTOZAP6OKHRWITGNBYHZUDAFNISCP7UF5IN7ACVQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:43:41.569296605 (60h7m51.291859645s) [noise=true , address=104.36.82.5:4837 , piece_id=L7XOW4TLACVBYVBB6POFHV7QNMIVR5D3URS2YY35Z7RMYQZPHO7Q, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:55784->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569302307 (60h7m51.291865367s) [noise=true , address=104.36.82.8:4972 , piece_id=ZLDAUEK4NYG6RAE4KEULX7WIDUHPG7HLNEKFLBFIEFTILFDEL6KQ, seg_pos={0 0}, upload=57680]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:41.569316373 (60h7m51.291879413s) [seg_pos={0 0}, upload=57680]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:43:41.805102594 (60h7m51.527665634s) [upload=57680]: Done uploading segments... Jul 26 23:43:41.805130116 (60h7m51.527693156s) [upload=57680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.23836489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.241817483s FAIL (2025-07-26 23:43:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.241839875s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244759941s FAIL (2025-07-26 23:43:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== O4NRXVDAJFQDMXRIJ3FCTTPORMKNLZZFKQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.935044304 (60h6m24.657607334s) [noise=true , address=79.127.205.233:4639 , piece_id=PBWVH2PPBJ34AVCS6DLOYZW5ARFXXRZBNIIZF4ZR4QY6IDINQ2HQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.935409392 (60h6m24.657972432s) [noise=true , address=79.127.220.101:4041 , piece_id=RT5QEDDYSX6EKKCBQPNTACJEQY56QCQUVSTX2S55Q5ISZSOD5DGQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.935497270 (60h6m24.6580603s) [noise=true , address=79.127.205.234:4604 , piece_id=PKXNVHTRDOFHQRLTBKDD2W6CWTHPXYAYPE5CF2KBKW3AH74424RA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.935734775 (60h6m24.658297805s) [noise=true , address=79.127.220.101:4048 , piece_id=3LDV2JIXSBYVA7RIOEB6SR3WZL7UOF7XGCCZTP27F5Y63N47WBOQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.936009270 (60h6m24.6585723s) [noise=true , address=104.36.82.55:4201 , piece_id=25OQRYXKFVZKTWEG2JEYGKAMT7NDVFEYWBW2OILZE43D2RKY4CSA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.936342307 (60h6m24.658905347s) [noise=true , address=156.146.43.225:4449 , piece_id=NWLA4EXOV5DAXXUXNFMOBNUNNIYCCA6RVUSFENOPYRZ7TADRTN2Q, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.936418763 (60h6m24.658981803s) [noise=true , address=79.127.205.246:4274 , piece_id=X7SQLQJW3JGVM6KKDE5IGM5BWBLX4DUBMWA6PFND72DXFWT7MFEQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.936700863 (60h6m24.659263893s) [noise=true , address=77.90.31.13:4643 , piece_id=E7IOB75VQS5RDOVEHC4SDG3J43DE5FABEL63WNLRS7TJTZY7OI7A, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.936931424 (60h6m24.659494454s) [noise=true , address=79.127.205.235:4682 , piece_id=YUYMK6YYBQD6YNJ36LBRMTOO5EN24DNPRCPTLIDXL7XPT6BHFIDA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.937083876 (60h6m24.659646916s) [noise=true , address=79.127.219.47:4048 , piece_id=JAILDZFAA7RGEROO42S4KGEYKLHGQFGQVQGECWFETCBKDFCSZOSQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.937301973 (60h6m24.659865013s) [noise=true , address=64.49.13.154:4927 , piece_id=65HPJKLMAEDFWYWEBSZSSOTYIUMY6NZTUH4LLAT6WSKCQGSZ2AVA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.937513076 (60h6m24.660076116s) [noise=true , address=104.36.82.46:4515 , piece_id=6T36BGOJ2X5N4D3JBZ4ADVY3KK2U6S2WFHFV34PLWCPU2ZNCL7KA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.937767985 (60h6m24.660331025s) [noise=true , address=104.36.82.54:4186 , piece_id=U54CHPTQ2EVTOIJO2P772B5MBO2Z5V4HTXLKFUFQI2X273SD2IOQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.937774196 (60h6m24.660337236s) [noise=true , address=104.36.82.41:4786 , piece_id=HNFLOEQEQA5BNWAKBG3EXVD2HFFM5RVJ33POEAAU2PGF26LNSH6A, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.938023453 (60h6m24.660586493s) [noise=true , address=104.36.82.52:4115 , piece_id=ZDNBGSTYYAR6ALWJNMTYQW4OMLWVGBQOPZNBJ4OM3MOZE7QAV7IQ, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.938212625 (60h6m24.660775665s) [noise=true , address=77.90.31.10:4523 , piece_id=UFWJG36NKZXHAO35OP2GHQVOUMLSXY3VAENPXYK4UKMWYO3O274A, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:14.938404803 (60h6m24.660967833s) [noise=true , address=104.36.82.53:4154 , piece_id=FXQ7WSOKUBR76DR6LQN2RJS4B24NREPLCBBT4PHDKBVYJJ3JYYRA, seg_pos={0 0}, upload=57666]: Uploading piece... Jul 26 23:42:15.972630127 (60h6m25.695193167s) [noise=true , address=156.146.43.225:4449 , piece_id=NWLA4EXOV5DAXXUXNFMOBNUNNIYCCA6RVUSFENOPYRZ7TADRTN2Q, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:16.040993677 (60h6m25.763556717s) [upload=57666]: Next returned nil segment Jul 26 23:42:16.041013375 (60h6m25.763576415s) [upload=57666]: Waiting for error group managing segments... Jul 26 23:42:16.326251982 (60h6m26.048815022s) [noise=true , address=64.49.13.154:4927 , piece_id=65HPJKLMAEDFWYWEBSZSSOTYIUMY6NZTUH4LLAT6WSKCQGSZ2AVA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.346455206 (60h6m26.069018236s) [noise=true , address=79.127.205.246:4274 , piece_id=X7SQLQJW3JGVM6KKDE5IGM5BWBLX4DUBMWA6PFND72DXFWT7MFEQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.346741464 (60h6m26.069304504s) [noise=true , address=79.127.205.234:4604 , piece_id=PKXNVHTRDOFHQRLTBKDD2W6CWTHPXYAYPE5CF2KBKW3AH74424RA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.389600427 (60h6m26.112163457s) [noise=true , address=79.127.205.245:4248 , piece_id=NQZ5GMFCLS4PV5JVFRO4NRXVDAJFQDMXRIJ3FCTTPORMKNLZZFKQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.425565248 (60h6m26.148128278s) [noise=true , address=79.127.205.235:4682 , piece_id=YUYMK6YYBQD6YNJ36LBRMTOO5EN24DNPRCPTLIDXL7XPT6BHFIDA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.895782515 (60h6m26.618345555s) [noise=true , address=79.127.205.233:4639 , piece_id=PBWVH2PPBJ34AVCS6DLOYZW5ARFXXRZBNIIZF4ZR4QY6IDINQ2HQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:16.939317401 (60h6m26.661880431s) [noise=true , address=79.127.219.47:4048 , piece_id=JAILDZFAA7RGEROO42S4KGEYKLHGQFGQVQGECWFETCBKDFCSZOSQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:18.813452344 (60h6m28.536015384s) [noise=true , address=79.127.220.101:4041 , piece_id=RT5QEDDYSX6EKKCBQPNTACJEQY56QCQUVSTX2S55Q5ISZSOD5DGQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:42:21.352214689 (60h6m31.074777729s) [noise=true , address=79.127.220.101:4048 , piece_id=3LDV2JIXSBYVA7RIOEB6SR3WZL7UOF7XGCCZTP27F5Y63N47WBOQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: <nil> Jul 26 23:43:13.849484351 (60h7m23.572047391s) [noise=true , address=104.36.82.41:4786 , piece_id=HNFLOEQEQA5BNWAKBG3EXVD2HFFM5RVJ33POEAAU2PGF26LNSH6A, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q, address:104.36.82.41:4786): protocol: rpc: dial tcp 104.36.82.41:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849572169 (60h7m23.572135209s) [noise=true , address=104.36.82.55:4201 , piece_id=25OQRYXKFVZKTWEG2JEYGKAMT7NDVFEYWBW2OILZE43D2RKY4CSA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: write tcp 188.166.225.170:58138->104.36.82.55:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849658063 (60h7m23.572221103s) [noise=true , address=77.90.31.10:4523 , piece_id=UFWJG36NKZXHAO35OP2GHQVOUMLSXY3VAENPXYK4UKMWYO3O274A, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849706737 (60h7m23.572269777s) [noise=true , address=104.36.82.54:4186 , piece_id=U54CHPTQ2EVTOIJO2P772B5MBO2Z5V4HTXLKFUFQI2X273SD2IOQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 188.166.225.170:46664->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849744560 (60h7m23.5723076s) [noise=true , address=104.36.82.46:4515 , piece_id=6T36BGOJ2X5N4D3JBZ4ADVY3KK2U6S2WFHFV34PLWCPU2ZNCL7KA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 188.166.225.170:38032->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849754188 (60h7m23.572317228s) [noise=true , address=104.36.82.53:4154 , piece_id=FXQ7WSOKUBR76DR6LQN2RJS4B24NREPLCBBT4PHDKBVYJJ3JYYRA, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:43:13.849784205 (60h7m23.572347235s) [noise=true , address=77.90.31.13:4643 , piece_id=E7IOB75VQS5RDOVEHC4SDG3J43DE5FABEL63WNLRS7TJTZY7OI7A, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA, address:77.90.31.13:4643): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:13.849791348 (60h7m23.572354379s) [noise=true , address=104.36.82.52:4115 , piece_id=ZDNBGSTYYAR6ALWJNMTYQW4OMLWVGBQOPZNBJ4OM3MOZE7QAV7IQ, seg_pos={0 0}, upload=57666]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:13.849827448 (60h7m23.572390488s) [seg_pos={0 0}, upload=57666]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:43:14.092098846 (60h7m23.814661886s) [upload=57666]: Done uploading segments... Jul 26 23:43:14.092123874 (60h7m23.814686914s) [upload=57666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.244779828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XX..XX.XX..XXX............XXXX.....XXX..X.....X......] duration: 29.391002408s FAIL (2025-07-26 23:43:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.391026935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.942575029s FAIL (2025-07-26 23:42:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.942607591s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X..] duration: 7.533406287s FAIL (2025-07-26 23:42:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.533426455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [....X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235934942s FAIL (2025-07-26 23:42:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 6 23:42:17.279742706 (60h6m27.002305747s) [upload=57668]: Done starting upload Jul 26 23:42:17.279847177 (60h6m27.002410277s) [upload=57668]: Uploading segments... Jul 26 23:42:17.279974871 (60h6m27.002537921s) [upload=57668]: Got next segment. Inline: false Jul 26 23:42:17.280026389 (60h6m27.002589429s) [upload=57668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:17.280035156 (60h6m27.002598206s) [upload=57668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:17.512742705 (60h6m27.235305766s) [seg_pos={0 0}, upload=57668]: Begin upload segment... Jul 26 23:42:17.512804884 (60h6m27.235367924s) [seg_pos={0 0}, upload=57668]: Done begin upload segment. Jul 26 23:42:17.512826726 (60h6m27.235389766s) [noise=true , address=104.36.82.16:4294 , piece_id=HZXONEUORUGVCUB7XNRCFK5XKYRQAUUXN3GQ64LEDCZHDIZAFIDQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.512894315 (60h6m27.235457355s) [noise=true , address=45.91.116.13:4155 , piece_id=4YCZVXPO6TP2XHXEF3YLXA4DVJ5PSCDPVLMV3VY7R3ZCOIXMHZNQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513163961 (60h6m27.235727001s) [noise=true , address=77.90.31.18:4805 , piece_id=XDKJXPHMWK7XAXANBIPYPUC6C22NQ2NIZVFR2Y4TG7K42LAX3AIQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513377560 (60h6m27.23594059s) [noise=true , address=104.36.82.4:4803 , piece_id=F7HYVQICYIYICNHJL3SOK3BIHDZQ3PM3AUB4CUX3KXEO5RQZN3ZQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513410964 (60h6m27.235973994s) [noise=true , address=104.36.82.11:4115 , piece_id=SIPBBGCWRCQHBO4PEBW4F5TTHT75HKMLCZKAYMUOQTWK3JLHJRSQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513708623 (60h6m27.236271663s) [noise=true , address=104.36.82.14:4222 , piece_id=GDBCOOJGDW3VQJUNHAROMA2SLWENNETTGOWUBUYSN4ZWBPB6SU4A, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.513753499 (60h6m27.236316529s) [noise=true , address=45.91.116.6:4975 , piece_id=GCHZT5PYMPMR3FZUWRSJEJSLYW63TRB63PLQHKJGFORHFMAR5NHA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514013717 (60h6m27.236576757s) [noise=true , address=104.36.82.13:4185 , piece_id=LLHIVKYSITYLRLW5A5FC75WEPA2M4WBVMV5B7MXYETGUOA6UICTA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514089142 (60h6m27.236652182s) [noise=true , address=104.36.82.2:4732 , piece_id=RGFMOCNUXXA7VCA3HJ3PJOQNZIS6KTEU7GF6F43PKHWXO2W4G5NA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514230993 (60h6m27.236794033s) [noise=true , address=104.36.82.3:4765 , piece_id=5JZKCPZFZLDY4SRH7NHCL7ML3M4UMLVMD5BHDDS5EWEDTQWV6GLA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514422880 (60h6m27.23698592s) [noise=true , address=45.91.116.7:4010 , piece_id=V4UYQF2M5N6SSZXIVEHTDWGLCUUTBHJXUSE46F6LPUQJVZXZI3PA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514453619 (60h6m27.237016659s) [noise=true , address=77.90.31.11:4553 , piece_id=UJ3ZHSDZMEEJOIBOMCZ6BOJ2RY5KDH6CCINSUE6CJIOPVKRJGDKA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514670043 (60h6m27.237233083s) [noise=true , address=77.90.31.21:4877 , piece_id=QUYBJ6CPBEVQMZVXVEIVZSFGHIFMY6AZLNK3FGVJGN4AJ6OEBMFQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.514780314 (60h6m27.237343354s) [noise=true , address=45.91.116.9:4261 , piece_id=LH4ICVQDJ2LURIATIFFLVPCDDYKBICI2WJODDHFOUXH2EA6OUDAQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.515038177 (60h6m27.237601207s) [noise=true , address=77.90.31.14:4663 , piece_id=VVNLCJV5QVQBRG47IVEBPRMVVQRBG4MELSDUYQGFYIHQTTDMP3XQ, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:17.517980547 (60h6m27.240543587s) [noise=true , address=104.36.82.15:4259 , piece_id=BREAEDTOOGFUDI7MFELAU5LFZFSRFNSGAIDSV3S6BAVT2CARVHSA, seg_pos={0 0}, upload=57668]: Uploading piece... Jul 26 23:42:19.269360754 (60h6m28.991923794s) [upload=57668]: Next returned nil segment Jul 26 23:42:19.269380342 (60h6m28.991943382s) [upload=57668]: Waiting for error group managing segments... Jul 26 23:42:19.545421040 (60h6m29.26798408s) [noise=true , address=45.91.116.13:4155 , piece_id=4YCZVXPO6TP2XHXEF3YLXA4DVJ5PSCDPVLMV3VY7R3ZCOIXMHZNQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.624565276 (60h6m29.347128316s) [noise=true , address=45.91.116.6:4975 , piece_id=GCHZT5PYMPMR3FZUWRSJEJSLYW63TRB63PLQHKJGFORHFMAR5NHA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.626840720 (60h6m29.34940376s) [noise=true , address=77.90.31.11:4553 , piece_id=UJ3ZHSDZMEEJOIBOMCZ6BOJ2RY5KDH6CCINSUE6CJIOPVKRJGDKA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.626891127 (60h6m29.349454157s) [noise=true , address=45.91.116.7:4010 , piece_id=V4UYQF2M5N6SSZXIVEHTDWGLCUUTBHJXUSE46F6LPUQJVZXZI3PA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.632696274 (60h6m29.355259324s) [noise=true , address=77.90.31.14:4663 , piece_id=VVNLCJV5QVQBRG47IVEBPRMVVQRBG4MELSDUYQGFYIHQTTDMP3XQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:19.852411615 (60h6m29.574974655s) [noise=true , address=45.91.116.9:4261 , piece_id=LH4ICVQDJ2LURIATIFFLVPCDDYKBICI2WJODDHFOUXH2EA6OUDAQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:21.206148120 (60h6m30.92871115s) [noise=true , address=77.90.31.18:4805 , piece_id=XDKJXPHMWK7XAXANBIPYPUC6C22NQ2NIZVFR2Y4TG7K42LAX3AIQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: <nil> Jul 26 23:42:46.640897359 (60h6m56.363460399s) [noise=true , address=104.36.82.14:4222 , piece_id=GDBCOOJGDW3VQJUNHAROMA2SLWENNETTGOWUBUYSN4ZWBPB6SU4A, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:46.640965670 (60h6m56.36352871s) [noise=true , address=104.36.82.11:4115 , piece_id=SIPBBGCWRCQHBO4PEBW4F5TTHT75HKMLCZKAYMUOQTWK3JLHJRSQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.640972293 (60h6m56.363535333s) [noise=true , address=104.36.82.4:4803 , piece_id=F7HYVQICYIYICNHJL3SOK3BIHDZQ3PM3AUB4CUX3KXEO5RQZN3ZQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:46.641009625 (60h6m56.363572665s) [noise=true , address=104.36.82.2:4732 , piece_id=RGFMOCNUXXA7VCA3HJ3PJOQNZIS6KTEU7GF6F43PKHWXO2W4G5NA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:35264->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641022629 (60h6m56.363585689s) [noise=true , address=104.36.82.16:4294 , piece_id=HZXONEUORUGVCUB7XNRCFK5XKYRQAUUXN3GQ64LEDCZHDIZAFIDQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:42:46.641041676 (60h6m56.363604716s) [noise=true , address=104.36.82.15:4259 , piece_id=BREAEDTOOGFUDI7MFELAU5LFZFSRFNSGAIDSV3S6BAVT2CARVHSA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:60152->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641093846 (60h6m56.363656886s) [noise=true , address=77.90.31.21:4877 , piece_id=QUYBJ6CPBEVQMZVXVEIVZSFGHIFMY6AZLNK3FGVJGN4AJ6OEBMFQ, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641111339 (60h6m56.363674379s) [noise=true , address=104.36.82.13:4185 , piece_id=LLHIVKYSITYLRLW5A5FC75WEPA2M4WBVMV5B7MXYETGUOA6UICTA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641124014 (60h6m56.363687054s) [noise=true , address=104.36.82.3:4765 , piece_id=5JZKCPZFZLDY4SRH7NHCL7ML3M4UMLVMD5BHDDS5EWEDTQWV6GLA, seg_pos={0 0}, upload=57668]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:46.641135906 (60h6m56.363698946s) [seg_pos={0 0}, upload=57668]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:42:46.875081343 (60h6m56.597644424s) [upload=57668]: Done uploading segments... Jul 26 23:42:46.875111722 (60h6m56.597674763s) [upload=57668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.235959067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237679062s FAIL (2025-07-26 23:42:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.237701834s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.833028084s FAIL (2025-07-26 23:42:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833053382s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X.....] duration: 27.667436505s FAIL (2025-07-26 23:42:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.667452515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.249836979s FAIL (2025-07-26 23:42:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== m12.532702764s) [seg_pos={0 0}, upload=57647]: Done begin upload segment. Jul 26 23:41:02.810161366 (60h5m12.532724406s) [noise=true , address=79.127.201.215:4350 , piece_id=HINDTR3MT4SHVNU4Y6EVA5BVA6BRZX3ILCLBYFWOVA62GGATU3NQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810277728 (60h5m12.532840768s) [noise=true , address=79.127.205.246:4275 , piece_id=E73QCW5QAG7ZZSDSZEKANVXIN2CIDIJBX556XHY3DWX6K7CF3TFQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810545791 (60h5m12.533108831s) [noise=true , address=79.127.220.97:4361 , piece_id=MHF6V6WHTLX6MG2SIZTMSAP6RA4A4ZRDKVHTS727Z5GCQM7G242Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810781161 (60h5m12.533344201s) [noise=true , address=104.36.82.53:4156 , piece_id=5F6UPSHANOQWQNL7SYHWFWBU7H7OTF2EZ2EHMOLVLRQKAW6VOMHQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.810806700 (60h5m12.53336973s) [noise=true , address=79.127.205.235:4685 , piece_id=JYAJ2NLIVVE4ILXE5YG7TM7ZQ43R44FJUHRSSCXVKCC2KLE3ANMA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811054224 (60h5m12.533617264s) [noise=true , address=104.36.82.42:4821 , piece_id=HLVQVVUPPELZTNMDZKAGYYFTJA7TSWO5BVV4EQBZP7B5SQWVHJ3Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811150719 (60h5m12.533713759s) [noise=true , address=64.49.13.140:4508 , piece_id=GDY3EEGJB2CIVBL3LZ5EDKDQ7KCC5Q3LJWBFABPNGS6ZH4NZ7LDA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811277701 (60h5m12.533840731s) [noise=true , address=104.36.82.52:4114 , piece_id=MOOXBMSDUSMZPAOSFCBUF6LHB52JT2S3H3VKMFTGYSIXSFIIBVCA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811453749 (60h5m12.534016789s) [noise=true , address=77.90.31.21:4895 , piece_id=PY4S7H5WDUBPFCDINMNNGWQV2RCLXEAAYKKEGVH3F7PLU6K666TA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811516639 (60h5m12.534079679s) [noise=true , address=104.36.82.56:4249 , piece_id=UZ76WXHIZQMVQ6VZW7VMS2AJUZPGGKMIZWXCOWGURD37PZKOWDDA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811744524 (60h5m12.534307564s) [noise=true , address=79.127.201.214:4321 , piece_id=2LBSADAWPVHIXH5KYBYX7NXQXLFUKGY4GJEM5EDXH6AQBPADIBNQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811756237 (60h5m12.534319267s) [noise=true , address=77.90.31.15:4714 , piece_id=3YFQ6PUG5H5LH6BNP4E44GJTMUJVIGO4OOYFE3VAGFMN6GODBP6Q, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.811978362 (60h5m12.534541402s) [noise=true , address=79.127.205.234:4607 , piece_id=W7UWXV6SWISPTD3OELN4UDWFIQR44T2BAQWTXRAFVZ3BYWINSMBA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812056752 (60h5m12.534619792s) [noise=true , address=104.36.82.44:4875 , piece_id=SYKIQDVGSDG7CZ5CTHWYXD24MPTXJWPVYK2Z7VWF5DI4BSF3FIJQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812269640 (60h5m12.53483268s) [noise=true , address=104.36.82.47:4531 , piece_id=BLQISDEF5BOHTR43CS7IOLRIMEUQSAVQMQOZG7S6IF2XBF646TSA, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812404498 (60h5m12.534967538s) [noise=true , address=79.127.219.47:4048 , piece_id=VVETUC72MUYLONSHR5BF4FBFVR7NAW66KPIPYME2WD75QJ7WFQ2A, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.812674213 (60h5m12.535237253s) [noise=true , address=77.90.31.12:4597 , piece_id=YA5XSKX4U3JJGAI7NIAU2MEYFV2GCNHDH6E3EVWRGLZIMWRQYJDQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:02.814084693 (60h5m12.536647723s) [noise=true , address=79.127.220.101:4041 , piece_id=UKGLEVF5VSYK6IQM52MTEDRHIPHNJDBA2VKD7MN7NRZXCES34MZQ, seg_pos={0 0}, upload=57647]: Uploading piece... Jul 26 23:41:03.830228169 (60h5m13.552791219s) [noise=true , address=79.127.220.97:4361 , piece_id=MHF6V6WHTLX6MG2SIZTMSAP6RA4A4ZRDKVHTS727Z5GCQM7G242Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 23:41:03.919165754 (60h5m13.641728784s) [upload=57647]: Next returned nil segment Jul 26 23:41:03.919181674 (60h5m13.641744704s) [upload=57647]: Waiting for error group managing segments... Jul 26 23:41:04.199928297 (60h5m13.922491337s) [noise=true , address=79.127.205.234:4607 , piece_id=W7UWXV6SWISPTD3OELN4UDWFIQR44T2BAQWTXRAFVZ3BYWINSMBA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.208241703 (60h5m13.930804743s) [noise=true , address=79.127.205.235:4685 , piece_id=JYAJ2NLIVVE4ILXE5YG7TM7ZQ43R44FJUHRSSCXVKCC2KLE3ANMA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.227137626 (60h5m13.949700666s) [noise=true , address=79.127.205.246:4275 , piece_id=E73QCW5QAG7ZZSDSZEKANVXIN2CIDIJBX556XHY3DWX6K7CF3TFQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.681758362 (60h5m14.404321402s) [noise=true , address=79.127.219.47:4048 , piece_id=VVETUC72MUYLONSHR5BF4FBFVR7NAW66KPIPYME2WD75QJ7WFQ2A, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.747406818 (60h5m14.469969868s) [noise=true , address=77.90.31.12:4597 , piece_id=YA5XSKX4U3JJGAI7NIAU2MEYFV2GCNHDH6E3EVWRGLZIMWRQYJDQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:04.844545002 (60h5m14.567108062s) [noise=true , address=77.90.31.15:4714 , piece_id=3YFQ6PUG5H5LH6BNP4E44GJTMUJVIGO4OOYFE3VAGFMN6GODBP6Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:05.652683113 (60h5m15.375246153s) [noise=true , address=79.127.220.101:4041 , piece_id=UKGLEVF5VSYK6IQM52MTEDRHIPHNJDBA2VKD7MN7NRZXCES34MZQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:07.253427507 (60h5m16.975990547s) [noise=true , address=79.127.201.214:4321 , piece_id=2LBSADAWPVHIXH5KYBYX7NXQXLFUKGY4GJEM5EDXH6AQBPADIBNQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:07.546097763 (60h5m17.268660803s) [noise=true , address=64.49.13.140:4508 , piece_id=GDY3EEGJB2CIVBL3LZ5EDKDQ7KCC5Q3LJWBFABPNGS6ZH4NZ7LDA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:41:27.601213531 (60h5m37.323776591s) [noise=true , address=79.127.201.215:4350 , piece_id=HINDTR3MT4SHVNU4Y6EVA5BVA6BRZX3ILCLBYFWOVA62GGATU3NQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: <nil> Jul 26 23:42:01.706639956 (60h6m11.429202996s) [noise=true , address=104.36.82.52:4114 , piece_id=MOOXBMSDUSMZPAOSFCBUF6LHB52JT2S3H3VKMFTGYSIXSFIIBVCA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:42:01.706679952 (60h6m11.429242992s) [noise=true , address=104.36.82.56:4249 , piece_id=UZ76WXHIZQMVQ6VZW7VMS2AJUZPGGKMIZWXCOWGURD37PZKOWDDA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 188.166.225.170:56940->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706721212 (60h6m11.429284252s) [noise=true , address=77.90.31.21:4895 , piece_id=PY4S7H5WDUBPFCDINMNNGWQV2RCLXEAAYKKEGVH3F7PLU6K666TA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: read tcp 188.166.225.170:41850->77.90.31.21:4895: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706767570 (60h6m11.42933061s) [noise=true , address=104.36.82.42:4821 , piece_id=HLVQVVUPPELZTNMDZKAGYYFTJA7TSWO5BVV4EQBZP7B5SQWVHJ3Q, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706792327 (60h6m11.429355367s) [noise=true , address=104.36.82.47:4531 , piece_id=BLQISDEF5BOHTR43CS7IOLRIMEUQSAVQMQOZG7S6IF2XBF646TSA, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706810943 (60h6m11.429373983s) [noise=true , address=104.36.82.53:4156 , piece_id=5F6UPSHANOQWQNL7SYHWFWBU7H7OTF2EZ2EHMOLVLRQKAW6VOMHQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: rpc: dial tcp 104.36.82.53:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:01.706881799 (60h6m11.429444839s) [noise=true , address=104.36.82.44:4875 , piece_id=SYKIQDVGSDG7CZ5CTHWYXD24MPTXJWPVYK2Z7VWF5DI4BSF3FIJQ, seg_pos={0 0}, upload=57647]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:104.36.82.44:4875): protocol: rpc: dial tcp 104.36.82.44:4875: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:42:01.706898761 (60h6m11.429461801s) [seg_pos={0 0}, upload=57647]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:42:01.954524682 (60h6m11.677087722s) [upload=57647]: Done uploading segments... Jul 26 23:42:01.954556963 (60h6m11.677120013s) [upload=57647]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.249850715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [.....X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.238261725s FAIL (2025-07-26 23:41:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== stItem_SegmentBegin Jul 26 23:41:24.496328373 (60h5m34.218891433s) [seg_pos={0 0}, upload=57654]: Begin upload segment... Jul 26 23:41:24.496381976 (60h5m34.218945016s) [seg_pos={0 0}, upload=57654]: Done begin upload segment. Jul 26 23:41:24.496413005 (60h5m34.218976075s) [noise=true , address=104.36.82.5:4840 , piece_id=PQEMYHQZFRYZJH7PGVQNFUSBFVRPFJQ4ED3SEJMQZGL3DACIHOWA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.496449575 (60h5m34.219012615s) [noise=true , address=45.91.116.8:4048 , piece_id=BJEJM22SFWOXJ7Z34UJATNP3EIUIGRR6RQL2EXRAHEXMIENO2FLQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.496930134 (60h5m34.219493174s) [noise=true , address=104.36.82.5:4837 , piece_id=BA5NDHGCADIZJWK2I2U5TEEOLRZP7SSNUOMPC2KV7YUNI3KM4DFA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497029315 (60h5m34.219592365s) [noise=true , address=104.36.82.14:4223 , piece_id=ZOTC3VW3RSTPR5IQGSDHD2PFOMEI3ZLSPB5AJUO7E3MJNM4EBHGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497339177 (60h5m34.219902217s) [noise=true , address=104.36.82.13:4186 , piece_id=PGQCAIPOUICCDLRB4VXOXPRZO3YL5HOJLRVL7TCUAWHPZDMVCGGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.497736147 (60h5m34.220299197s) [noise=true , address=45.91.116.10:4083 , piece_id=XO3G5WPRNPIJ6IYYP7R73TGJSILGURNGFUEZ66K56RXZWD7WYPZA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498120503 (60h5m34.220683553s) [noise=true , address=104.36.82.6:4875 , piece_id=QD7QKQEFENY3MR754BBBOKPYKGPJA6F3UHND5KGMJO6D7JQTCZEQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498451135 (60h5m34.221014175s) [noise=true , address=104.36.82.9:4005 , piece_id=AK7VG6ZHPQGMA2X3IUGU46F7GXLL57FGNHMRN2HP24K4UKGJNTGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.498751699 (60h5m34.221314739s) [noise=true , address=104.36.82.8:4971 , piece_id=MXFGKXD5IYSV3JVL5A7WHINHYTRR7EEHHI7VLSN3ZMW5HVUOPUUA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499076571 (60h5m34.221639611s) [noise=true , address=45.91.116.7:4009 , piece_id=ULX6U7BOCMBCUOMYW7JA37BGX7AKA3FYTLAV4Z65SXRV7Y5ZXAYQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499352048 (60h5m34.221915088s) [noise=true , address=104.36.82.11:4115 , piece_id=BCIRCPFTDOSPSPA7ELRYX47IKZ6OZMRGV2FTXWIRBEKPGDQ2IUGQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499666310 (60h5m34.22222935s) [noise=true , address=104.36.82.9:4007 , piece_id=ZVHQ5WMYOJWG7CI3I6TF2O65PARCRY5KJ2WJTXQHXGPM53UEEPBA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.499973137 (60h5m34.222536177s) [noise=true , address=45.91.116.4:4937 , piece_id=RWLIKN33SREMPPV2SREPCFQZO57N3OM7IUUCVGI2YCF754QFTUAA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500298129 (60h5m34.222861169s) [noise=true , address=45.91.116.6:4974 , piece_id=WADUCHLS6HELMRISN76TN4X4WJKRRLUTTVS4T7BJVUJM3Y3F3CBA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500594335 (60h5m34.223157375s) [noise=true , address=104.36.82.13:4187 , piece_id=FZ7SV2V2ZL3YZ2ADXAJ65L5SVHNV7DOFWKLIGHC5W3H76W45DBIA, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:24.500904559 (60h5m34.223467599s) [noise=true , address=45.91.116.12:4225 , piece_id=LOV5ZIIUNRXYZ5GXFAQTTOORZHO7IL7W4KJ5SNEGSRLO535CPDFQ, seg_pos={0 0}, upload=57654]: Uploading piece... Jul 26 23:41:26.259926216 (60h5m35.982489246s) [upload=57654]: Next returned nil segment Jul 26 23:41:26.259943168 (60h5m35.982506208s) [upload=57654]: Waiting for error group managing segments... Jul 26 23:41:26.535548654 (60h5m36.258111694s) [noise=true , address=45.91.116.7:4009 , piece_id=ULX6U7BOCMBCUOMYW7JA37BGX7AKA3FYTLAV4Z65SXRV7Y5ZXAYQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.591695871 (60h5m36.314258911s) [noise=true , address=45.91.116.4:4937 , piece_id=RWLIKN33SREMPPV2SREPCFQZO57N3OM7IUUCVGI2YCF754QFTUAA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.620833340 (60h5m36.34339637s) [noise=true , address=45.91.116.12:4225 , piece_id=LOV5ZIIUNRXYZ5GXFAQTTOORZHO7IL7W4KJ5SNEGSRLO535CPDFQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.621147070 (60h5m36.34371011s) [noise=true , address=45.91.116.6:4974 , piece_id=WADUCHLS6HELMRISN76TN4X4WJKRRLUTTVS4T7BJVUJM3Y3F3CBA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.628699220 (60h5m36.35126226s) [noise=true , address=45.91.116.8:4048 , piece_id=BJEJM22SFWOXJ7Z34UJATNP3EIUIGRR6RQL2EXRAHEXMIENO2FLQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:26.765689543 (60h5m36.488252583s) [noise=true , address=45.91.116.10:4083 , piece_id=XO3G5WPRNPIJ6IYYP7R73TGJSILGURNGFUEZ66K56RXZWD7WYPZA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: <nil> Jul 26 23:41:53.619874012 (60h6m3.342437052s) [noise=true , address=104.36.82.14:4223 , piece_id=ZOTC3VW3RSTPR5IQGSDHD2PFOMEI3ZLSPB5AJUO7E3MJNM4EBHGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:36664->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619917936 (60h6m3.342480976s) [noise=true , address=104.36.82.9:4005 , piece_id=AK7VG6ZHPQGMA2X3IUGU46F7GXLL57FGNHMRN2HP24K4UKGJNTGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619947413 (60h6m3.342510453s) [noise=true , address=104.36.82.8:4971 , piece_id=MXFGKXD5IYSV3JVL5A7WHINHYTRR7EEHHI7VLSN3ZMW5HVUOPUUA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:42816->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.619975226 (60h6m3.342538266s) [noise=true , address=104.36.82.13:4187 , piece_id=FZ7SV2V2ZL3YZ2ADXAJ65L5SVHNV7DOFWKLIGHC5W3H76W45DBIA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620005053 (60h6m3.342568093s) [noise=true , address=104.36.82.5:4837 , piece_id=BA5NDHGCADIZJWK2I2U5TEEOLRZP7SSNUOMPC2KV7YUNI3KM4DFA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:46530->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620032275 (60h6m3.342595305s) [noise=true , address=104.36.82.6:4875 , piece_id=QD7QKQEFENY3MR754BBBOKPYKGPJA6F3UHND5KGMJO6D7JQTCZEQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:48892->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620059908 (60h6m3.342622948s) [noise=true , address=104.36.82.11:4115 , piece_id=BCIRCPFTDOSPSPA7ELRYX47IKZ6OZMRGV2FTXWIRBEKPGDQ2IUGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620086899 (60h6m3.34264994s) [noise=true , address=104.36.82.13:4186 , piece_id=PGQCAIPOUICCDLRB4VXOXPRZO3YL5HOJLRVL7TCUAWHPZDMVCGGQ, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:39234->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620114392 (60h6m3.342677422s) [noise=true , address=104.36.82.9:4007 , piece_id=ZVHQ5WMYOJWG7CI3I6TF2O65PARCRY5KJ2WJTXQHXGPM53UEEPBA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620143828 (60h6m3.342706868s) [noise=true , address=104.36.82.5:4840 , piece_id=PQEMYHQZFRYZJH7PGVQNFUSBFVRPFJQ4ED3SEJMQZGL3DACIHOWA, seg_pos={0 0}, upload=57654]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:32910->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:53.620359511 (60h6m3.342922551s) [seg_pos={0 0}, upload=57654]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:41:53.856061540 (60h6m3.57862458s) [upload=57654]: Done uploading segments... Jul 26 23:41:53.856086668 (60h6m3.578649708s) [upload=57654]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.238276393s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X.] duration: 7.70407651s FAIL (2025-07-26 23:41:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.704102448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [......X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.233307354s FAIL (2025-07-26 23:41:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== l 26 23:40:53.798921398 (60h5m3.521484438s) [upload=57643]: Starting upload Jul 26 23:40:53.798984470 (60h5m3.52154751s) [upload=57643]: Done starting upload Jul 26 23:40:53.799053832 (60h5m3.521616872s) [upload=57643]: Uploading segments... Jul 26 23:40:53.799162290 (60h5m3.52172533s) [upload=57643]: Got next segment. Inline: false Jul 26 23:40:53.799196314 (60h5m3.521759354s) [upload=57643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:40:53.799209410 (60h5m3.52177245s) [upload=57643]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:40:54.065144082 (60h5m3.787707132s) [seg_pos={0 0}, upload=57643]: Begin upload segment... Jul 26 23:40:54.065204337 (60h5m3.787767377s) [seg_pos={0 0}, upload=57643]: Done begin upload segment. Jul 26 23:40:54.065233834 (60h5m3.787796874s) [noise=true , address=45.91.116.10:4083 , piece_id=XIPEAMLQ7XASULSDU4IYGFZODUMXKWC4HIBDTSF57ECREBALVMYA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065288709 (60h5m3.787851749s) [noise=true , address=104.36.82.13:4187 , piece_id=G3VQ6UIIWQXTPFZYFGQ25HWODTBAQZYWVVAJWOLOS5ALNJGDAHBA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065838520 (60h5m3.788401581s) [noise=true , address=45.91.116.12:4225 , piece_id=W3AJYCVJZ5X7O3M3ERKIUJRVBKQBZ5JPBQI2OQP2ZUDBGDMGKP3Q, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.065924976 (60h5m3.788488016s) [noise=true , address=45.91.116.6:4973 , piece_id=HSQIBXKQNP3LTRF6SPEYXJVUHLT3Q3EB6XSOPA2AQ7N3SID73AFA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066242133 (60h5m3.788805173s) [noise=true , address=104.36.82.5:4839 , piece_id=4NVOW5SCB6357X4OC2LBN5UDKN73JD3RACG6KTFG7LUC3ECD5MEQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066343667 (60h5m3.788906707s) [noise=true , address=104.36.82.9:4007 , piece_id=P4ZZAX4H7SNII47EWOF4JZCZ2CCTGXJFXHVBFXYFF5Q6KXHRR6WQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066682385 (60h5m3.789245425s) [noise=true , address=45.91.116.11:4118 , piece_id=7QQIPZSUOTQ6SULT4UP2CT5JXZE2DGTOWKCEJ54KLYD573GPFDSQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.066779541 (60h5m3.789342581s) [noise=true , address=104.36.82.8:4969 , piece_id=YEXVSD6POLNBCC6QWSBVAT2CBCPPRAUOBTOT2MJXQQ3SNFTBTIJQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067030912 (60h5m3.789593952s) [noise=true , address=104.36.82.4:4802 , piece_id=SRNVGYMPYK7BJBX2MZGZ3CRTZ6O4TCY6TTHDSUUIXZHCNLOBLMFQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067184335 (60h5m3.789747376s) [noise=true , address=104.36.82.16:4293 , piece_id=VXLOQOHR7ZOX6QV4K252TARFU7ERB7WSTLO6PQR3LVNK3PWOSNSQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067375692 (60h5m3.789938732s) [noise=true , address=104.36.82.7:4912 , piece_id=C3NV4RSZGQZYX7WOBXL3QQJPFNLGVYFQVYBOAU2ZWE5EBZGXIF5A, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067563241 (60h5m3.790126281s) [noise=true , address=104.36.82.10:4078 , piece_id=5KZ3MYPM3QI6DEHKSFRVCMHBV3XLTBRDSPSTBHOFUTCJFHVTLWZQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067733887 (60h5m3.790296927s) [noise=true , address=45.91.116.8:4045 , piece_id=K6EU6IWY2C4JNFKE3GEHO2TXITUR4NDR36YNQ4DT574N3TDVX2YA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.067955040 (60h5m3.79051808s) [noise=true , address=45.91.116.4:4938 , piece_id=ZBHBICLFFLQZCUU5M6LPN7RBQR2ABGVC6XWID3ZNXF5KTJ22LBDQ, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.068047907 (60h5m3.790610947s) [noise=true , address=45.91.116.8:4047 , piece_id=S5Z6E7DB7LLJLZ2TI6ZMHL6OI4ZQYNWHDZEPIN2H2IJ4T2BZF3JA, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:54.068357320 (60h5m3.79092037s) [noise=true , address=45.91.116.13:4155 , piece_id=N6S2QTH5N3XXOJ3VL6AXMVP37MALI7MNAAJCMUZ5FLUTKE564M3A, seg_pos={0 0}, upload=57643]: Uploading piece... Jul 26 23:40:55.813245504 (60h5m5.535808544s) [upload=57643]: Next returned nil segment Jul 26 23:40:55.813263219 (60h5m5.535826259s) [upload=57643]: Waiting for error group managing segments... Jul 26 23:40:56.090611589 (60h5m5.81317463s) [noise=true , address=45.91.116.8:4045 , piece_id=K6EU6IWY2C4JNFKE3GEHO2TXITUR4NDR36YNQ4DT574N3TDVX2YA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.095243321 (60h5m5.817806361s) [noise=true , address=45.91.116.10:4083 , piece_id=XIPEAMLQ7XASULSDU4IYGFZODUMXKWC4HIBDTSF57ECREBALVMYA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.097127957 (60h5m5.819690997s) [noise=true , address=45.91.116.13:4155 , piece_id=N6S2QTH5N3XXOJ3VL6AXMVP37MALI7MNAAJCMUZ5FLUTKE564M3A, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.176113910 (60h5m5.898676951s) [noise=true , address=45.91.116.8:4047 , piece_id=S5Z6E7DB7LLJLZ2TI6ZMHL6OI4ZQYNWHDZEPIN2H2IJ4T2BZF3JA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.176198943 (60h5m5.898761983s) [noise=true , address=45.91.116.12:4225 , piece_id=W3AJYCVJZ5X7O3M3ERKIUJRVBKQBZ5JPBQI2OQP2ZUDBGDMGKP3Q, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.202081505 (60h5m5.924644575s) [noise=true , address=45.91.116.4:4938 , piece_id=ZBHBICLFFLQZCUU5M6LPN7RBQR2ABGVC6XWID3ZNXF5KTJ22LBDQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.227513815 (60h5m5.950076855s) [noise=true , address=45.91.116.6:4973 , piece_id=HSQIBXKQNP3LTRF6SPEYXJVUHLT3Q3EB6XSOPA2AQ7N3SID73AFA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:40:56.257081847 (60h5m5.979644887s) [noise=true , address=45.91.116.11:4118 , piece_id=7QQIPZSUOTQ6SULT4UP2CT5JXZE2DGTOWKCEJ54KLYD573GPFDSQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: <nil> Jul 26 23:41:23.158146731 (60h5m32.880709771s) [noise=true , address=104.36.82.7:4912 , piece_id=C3NV4RSZGQZYX7WOBXL3QQJPFNLGVYFQVYBOAU2ZWE5EBZGXIF5A, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158190023 (60h5m32.880753063s) [noise=true , address=104.36.82.8:4969 , piece_id=YEXVSD6POLNBCC6QWSBVAT2CBCPPRAUOBTOT2MJXQQ3SNFTBTIJQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:36076->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158205493 (60h5m32.880768533s) [noise=true , address=104.36.82.13:4187 , piece_id=G3VQ6UIIWQXTPFZYFGQ25HWODTBAQZYWVVAJWOLOS5ALNJGDAHBA, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 188.166.225.170:51644->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158245880 (60h5m32.88080892s) [noise=true , address=104.36.82.5:4839 , piece_id=4NVOW5SCB6357X4OC2LBN5UDKN73JD3RACG6KTFG7LUC3ECD5MEQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158274154 (60h5m32.880837194s) [noise=true , address=104.36.82.10:4078 , piece_id=5KZ3MYPM3QI6DEHKSFRVCMHBV3XLTBRDSPSTBHOFUTCJFHVTLWZQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158282551 (60h5m32.880845581s) [noise=true , address=104.36.82.9:4007 , piece_id=P4ZZAX4H7SNII47EWOF4JZCZ2CCTGXJFXHVBFXYFF5Q6KXHRR6WQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:60692->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158292049 (60h5m32.880855089s) [noise=true , address=104.36.82.16:4293 , piece_id=VXLOQOHR7ZOX6QV4K252TARFU7ERB7WSTLO6PQR3LVNK3PWOSNSQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158309132 (60h5m32.880872172s) [noise=true , address=104.36.82.4:4802 , piece_id=SRNVGYMPYK7BJBX2MZGZ3CRTZ6O4TCY6TTHDSUUIXZHCNLOBLMFQ, seg_pos={0 0}, upload=57643]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:57370->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:23.158325593 (60h5m32.880888623s) [seg_pos={0 0}, upload=57643]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:41:23.389168695 (60h5m33.111731735s) [upload=57643]: Done uploading segments... Jul 26 23:41:23.389195697 (60h5m33.111758727s) [upload=57643]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.233327272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.23265294s FAIL (2025-07-26 23:41:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232713516s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.831603045s FAIL (2025-07-26 23:41:13 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.831624766s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X....] duration: 38.739162725s FAIL (2025-07-26 23:41:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.739183824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236461143s FAIL (2025-07-26 23:41:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== next segment. Inline: false Jul 26 23:40:01.891301178 (60h4m11.613864228s) [upload=57631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:40:01.891344101 (60h4m11.613907151s) [upload=57631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:40:02.334018641 (60h4m12.056581701s) [seg_pos={0 0}, upload=57631]: Begin upload segment... Jul 26 23:40:02.334094336 (60h4m12.056657386s) [seg_pos={0 0}, upload=57631]: Done begin upload segment. Jul 26 23:40:02.334124964 (60h4m12.056688014s) [noise=true , address=79.127.201.215:4358 , piece_id=H3CFX2TNM5WOEP6OO63HNH3TXTHH6I36AUZMSN2FIS7OWBVL62AA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334598690 (60h4m12.05716174s) [noise=true , address=64.49.13.140:4514 , piece_id=HB5MLNZGGJBJFQUMI7C7XDAMJWFYYSUK55Q5SSRJ3YPRKSHJHIMA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334817048 (60h4m12.057380088s) [noise=true , address=79.127.219.51:4196 , piece_id=NKFK5YHDYJDGRT76JDACGPCEEMHNNMVDW4JCDM2WQ4Q2XXF77CQQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.334970051 (60h4m12.057533101s) [noise=true , address=104.36.82.55:4228 , piece_id=JP5LCG653OM4IBJZYK2E7LXLFPD4MD4LGLEY74NOYMKR2ORASTCQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335327766 (60h4m12.057890816s) [noise=true , address=79.127.205.235:4691 , piece_id=HBH3UHRNUF2WUOHIMDN2HF6YDK7USXRM66KQZVX53CI2LMDNXJQA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335352513 (60h4m12.057915563s) [noise=true , address=104.36.82.53:4140 , piece_id=GD422AISDOMSTCWJ33TAOQYYP2YOPEVCEDITIF5DEIGBN3RGXLGQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335670842 (60h4m12.058233892s) [noise=true , address=104.36.82.56:4259 , piece_id=CRKOTNKL4DZEYFVZIFMMRTI6YQP5IUN32IU3UM7TI254PHDM54EA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.335681172 (60h4m12.058244212s) [noise=true , address=64.49.13.138:4483 , piece_id=MSQ2TKY75KQQ6IWOUOBF23Z7UZY6IUSEIHL3SEBQJI3VCC6KYGPQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336019499 (60h4m12.058582549s) [noise=true , address=64.49.13.138:4482 , piece_id=JG6NCXMZZ65SPWD6QPIVHQXSJVHCHVX5O3ARDBZNBEUQ35ZZMVNA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336288354 (60h4m12.058851404s) [noise=true , address=77.90.31.12:4611 , piece_id=W2HGHPCQ3SCRZXAMDGKPMTKQ5GUW6MBYBEOKGKMDBJIBLRORIV4Q, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336492574 (60h4m12.059055624s) [noise=true , address=79.127.205.245:4249 , piece_id=GUUNLUR2FAES7EYCY3LT4XB3EA4JVV7ZDGS37T7XLT6NOZTUFGBA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.336667600 (60h4m12.05923065s) [noise=true , address=104.36.82.47:4548 , piece_id=GS7HYU3RGLABF7RRQNKUSESTYR2GUDAYNTSAM6JRQFR2OOESDDVQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337004233 (60h4m12.059567273s) [noise=true , address=79.127.205.234:4618 , piece_id=C7Q5X2K4GLJRWSBLY4EK7ZKKUSPQX43EMGW7XXFVFYMEPZAAISZA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337404198 (60h4m12.059967238s) [noise=true , address=104.36.82.54:4170 , piece_id=EKAXD6L76FPRQMPNTDLYYUKETWVCGIIKYH5UKF3KJPFG5HURKEKQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.337693081 (60h4m12.060256121s) [noise=true , address=79.127.219.38:4009 , piece_id=TI6JLARWL77E2TSAPNXOGCUWCHAW42746NQHJLA7EERSAEAXFKYQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338041528 (60h4m12.060604578s) [noise=true , address=104.36.82.20:4447 , piece_id=GRDH4FZBYAEOIYZ7SKG2QJUCBFMPIF4PHXF3OAEZ2CT2TDK7XOTQ, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338435281 (60h4m12.060998321s) [noise=true , address=45.91.116.14:4206 , piece_id=4W4IYEOTJ2ECZR67RFJUB4CPDEAFAKADKBSVS5B3SGYZZVTE6GPA, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:02.338792094 (60h4m12.061355144s) [noise=true , address=45.91.116.12:4230 , piece_id=AX3DJTKNF7TGIIOTZZGIPZDOR7CUJVI73IRRLGCOFAHZ4AB35Y3A, seg_pos={0 0}, upload=57631]: Uploading piece... Jul 26 23:40:03.435253225 (60h4m13.157816275s) [upload=57631]: Next returned nil segment Jul 26 23:40:03.435271992 (60h4m13.157835032s) [upload=57631]: Waiting for error group managing segments... Jul 26 23:40:03.713687615 (60h4m13.436250665s) [noise=true , address=79.127.205.245:4249 , piece_id=GUUNLUR2FAES7EYCY3LT4XB3EA4JVV7ZDGS37T7XLT6NOZTUFGBA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.724736384 (60h4m13.447299424s) [noise=true , address=79.127.205.235:4691 , piece_id=HBH3UHRNUF2WUOHIMDN2HF6YDK7USXRM66KQZVX53CI2LMDNXJQA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.745523066 (60h4m13.468086106s) [noise=true , address=64.49.13.140:4514 , piece_id=HB5MLNZGGJBJFQUMI7C7XDAMJWFYYSUK55Q5SSRJ3YPRKSHJHIMA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:03.776899468 (60h4m13.499462518s) [noise=true , address=79.127.205.234:4618 , piece_id=C7Q5X2K4GLJRWSBLY4EK7ZKKUSPQX43EMGW7XXFVFYMEPZAAISZA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.196053668 (60h4m13.918616718s) [noise=true , address=79.127.219.51:4196 , piece_id=NKFK5YHDYJDGRT76JDACGPCEEMHNNMVDW4JCDM2WQ4Q2XXF77CQQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.240435483 (60h4m13.962998533s) [noise=true , address=79.127.219.38:4009 , piece_id=TI6JLARWL77E2TSAPNXOGCUWCHAW42746NQHJLA7EERSAEAXFKYQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:04.344412083 (60h4m14.066975123s) [noise=true , address=77.90.31.12:4611 , piece_id=W2HGHPCQ3SCRZXAMDGKPMTKQ5GUW6MBYBEOKGKMDBJIBLRORIV4Q, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:05.211183609 (60h4m14.933746649s) [noise=true , address=64.49.13.138:4482 , piece_id=JG6NCXMZZ65SPWD6QPIVHQXSJVHCHVX5O3ARDBZNBEUQ35ZZMVNA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:08.895379238 (60h4m18.617942268s) [noise=true , address=64.49.13.138:4483 , piece_id=MSQ2TKY75KQQ6IWOUOBF23Z7UZY6IUSEIHL3SEBQJI3VCC6KYGPQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:10.854276141 (60h4m20.576839181s) [noise=true , address=45.91.116.12:4230 , piece_id=AX3DJTKNF7TGIIOTZZGIPZDOR7CUJVI73IRRLGCOFAHZ4AB35Y3A, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:11.901007675 (60h4m21.623570715s) [noise=true , address=45.91.116.14:4206 , piece_id=4W4IYEOTJ2ECZR67RFJUB4CPDEAFAKADKBSVS5B3SGYZZVTE6GPA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:40:42.400223133 (60h4m52.122786173s) [noise=true , address=79.127.201.215:4358 , piece_id=H3CFX2TNM5WOEP6OO63HNH3TXTHH6I36AUZMSN2FIS7OWBVL62AA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: <nil> Jul 26 23:41:01.238787306 (60h5m10.961350356s) [noise=true , address=104.36.82.53:4140 , piece_id=GD422AISDOMSTCWJ33TAOQYYP2YOPEVCEDITIF5DEIGBN3RGXLGQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: rpc: dial tcp 104.36.82.53:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.238869393 (60h5m10.961432433s) [noise=true , address=104.36.82.56:4259 , piece_id=CRKOTNKL4DZEYFVZIFMMRTI6YQP5IUN32IU3UM7TI254PHDM54EA, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:41:01.238917414 (60h5m10.961480454s) [noise=true , address=104.36.82.47:4548 , piece_id=GS7HYU3RGLABF7RRQNKUSESTYR2GUDAYNTSAM6JRQFR2OOESDDVQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.238967710 (60h5m10.96153075s) [noise=true , address=104.36.82.20:4447 , piece_id=GRDH4FZBYAEOIYZ7SKG2QJUCBFMPIF4PHXF3OAEZ2CT2TDK7XOTQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1Li8ZwrWGPx5Y478pYLwnWJ7N2x3dbXgwGaUGWVtVhvv61EnTs, address:104.36.82.20:4447): protocol: write tcp 188.166.225.170:51350->104.36.82.20:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239026923 (60h5m10.961589964s) [noise=true , address=104.36.82.54:4170 , piece_id=EKAXD6L76FPRQMPNTDLYYUKETWVCGIIKYH5UKF3KJPFG5HURKEKQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: rpc: dial tcp 104.36.82.54:4170: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239067111 (60h5m10.961630151s) [noise=true , address=104.36.82.55:4228 , piece_id=JP5LCG653OM4IBJZYK2E7LXLFPD4MD4LGLEY74NOYMKR2ORASTCQ, seg_pos={0 0}, upload=57631]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 188.166.225.170:40190->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:01.239095826 (60h5m10.961658866s) [seg_pos={0 0}, upload=57631]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:41:01.473296372 (60h5m11.195859422s) [upload=57631]: Done uploading segments... Jul 26 23:41:01.473375273 (60h5m11.195938323s) [upload=57631]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.236478415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X] duration: 30.000445402s FAIL (2025-07-26 23:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000462215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235260476s FAIL (2025-07-26 23:40:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235283051s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.......X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232650781s FAIL (2025-07-26 23:40:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== [upload=57629]: Done starting upload Jul 26 23:39:54.613218404 (60h4m4.335781494s) [upload=57629]: Uploading segments... Jul 26 23:39:54.613596457 (60h4m4.336159538s) [upload=57629]: Got next segment. Inline: false Jul 26 23:39:54.613683073 (60h4m4.336246163s) [upload=57629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:54.613691819 (60h4m4.33625489s) [upload=57629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:54.847068078 (60h4m4.569631118s) [seg_pos={0 0}, upload=57629]: Begin upload segment... Jul 26 23:39:54.847140146 (60h4m4.569703186s) [seg_pos={0 0}, upload=57629]: Done begin upload segment. Jul 26 23:39:54.847171676 (60h4m4.569734716s) [noise=true , address=104.36.82.12:4150 , piece_id=OID3AOLTTEGHC7FI4ABETMTI5TIM7IALBJHE3IX2DFBA756LAMQQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.847501036 (60h4m4.570064076s) [noise=true , address=104.36.82.16:4295 , piece_id=T2Z3M7JYTMCT3X2AHFDMHCWBV23T33BKOUMUJY4S4THMBGYQDROQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.847747057 (60h4m4.570310087s) [noise=true , address=45.91.116.7:4011 , piece_id=X6LZCMNYIWDQF3FCZXE56MWYZEMBJ4BFZGXDPDG4HDPCQORZYYPQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848023155 (60h4m4.570586195s) [noise=true , address=104.36.82.7:4912 , piece_id=HWKVVPMIBSYGOVFRDXZS2NCSIWQZH4LJUJ3V3SDME2QDOR2TQM6A, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848387672 (60h4m4.570950713s) [noise=true , address=77.90.31.18:4806 , piece_id=EZ2YVAHV6AA7OZWKHQG6J5ULGYXMQXLVWVHRASRLPQ2LDJG5IA7Q, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.848730528 (60h4m4.571293568s) [noise=true , address=45.91.116.13:4153 , piece_id=YKW7X3J77MD255KUBGSBPKIT66CHMTTNIHCYIUVYQMLPR6Z2P5PQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849008911 (60h4m4.571571951s) [noise=true , address=104.36.82.5:4840 , piece_id=X35OIBIFJVBKWZ2TS4MBCA6J4GKYICZTMH5ZOEODF6IDONHYIAAA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849216578 (60h4m4.571779618s) [noise=true , address=45.91.116.6:4974 , piece_id=ZPCKVLCI4TJVXJHCL24Z5NKTD775HFBUND4UEMHHHZN675FY4HHQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849531390 (60h4m4.57209443s) [noise=true , address=45.91.116.5:4297 , piece_id=IOAI7IFMLDABT2H7RSCR3TQH7QA7PVTM4FSSJ7KLEI5ELHR337PA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849739569 (60h4m4.572302599s) [noise=true , address=104.36.82.2:4732 , piece_id=NHZWL4O44AKJ2GIZ6KDO5KM66JIM42TZTXZR3VWJKIHFLWQJ4GCQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.849871912 (60h4m4.572434952s) [noise=true , address=104.36.82.9:4007 , piece_id=N2M5H3LFJWZKMGKRCHLANERL2MBT7LERMRECWVKBPEQB2GCVGPIA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850145285 (60h4m4.572708325s) [noise=true , address=45.91.116.10:4081 , piece_id=ADABOJYSIWFVQNDFEKBJI3DKJAETE2NWAB36IM7NKKLL3JYSKC5A, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850517116 (60h4m4.573080156s) [noise=true , address=104.36.82.14:4223 , piece_id=DEWIE4BGTOCEFQ3MRMQTPL34K7MPH4QNO3OYKINH4DXSNJJR7CIA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.850860484 (60h4m4.573423534s) [noise=true , address=104.36.82.3:4767 , piece_id=K3J4WV5GD2MYO2ZZ6MKDTOGM3PBKJENJ2EGHYDFI55BFAZW5V7IQ, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.851112666 (60h4m4.573675706s) [noise=true , address=77.90.31.13:4627 , piece_id=VR7ZYG7CAY26LNQB4KXSLZ67AQ3YCNYZSDSPQUAJH4VAUUWMSB4Q, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:54.851328700 (60h4m4.57389174s) [noise=true , address=45.91.116.4:4940 , piece_id=3BZU4G5NW3OXGY7ASUPAS4CT5MBY5D7LSZON6C5HRRCOFJXURNSA, seg_pos={0 0}, upload=57629]: Uploading piece... Jul 26 23:39:56.571267441 (60h4m6.293830492s) [upload=57629]: Next returned nil segment Jul 26 23:39:56.571287309 (60h4m6.293850349s) [upload=57629]: Waiting for error group managing segments... Jul 26 23:39:56.861708441 (60h4m6.584271471s) [noise=true , address=45.91.116.6:4974 , piece_id=ZPCKVLCI4TJVXJHCL24Z5NKTD775HFBUND4UEMHHHZN675FY4HHQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.869633655 (60h4m6.592196695s) [noise=true , address=45.91.116.4:4940 , piece_id=3BZU4G5NW3OXGY7ASUPAS4CT5MBY5D7LSZON6C5HRRCOFJXURNSA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.962593204 (60h4m6.685156244s) [noise=true , address=45.91.116.13:4153 , piece_id=YKW7X3J77MD255KUBGSBPKIT66CHMTTNIHCYIUVYQMLPR6Z2P5PQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.963153938 (60h4m6.685716978s) [noise=true , address=77.90.31.18:4806 , piece_id=EZ2YVAHV6AA7OZWKHQG6J5ULGYXMQXLVWVHRASRLPQ2LDJG5IA7Q, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:56.965557095 (60h4m6.688120125s) [noise=true , address=45.91.116.7:4011 , piece_id=X6LZCMNYIWDQF3FCZXE56MWYZEMBJ4BFZGXDPDG4HDPCQORZYYPQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:57.045039820 (60h4m6.76760286s) [noise=true , address=45.91.116.10:4081 , piece_id=ADABOJYSIWFVQNDFEKBJI3DKJAETE2NWAB36IM7NKKLL3JYSKC5A, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:57.083542838 (60h4m6.806105878s) [noise=true , address=45.91.116.5:4297 , piece_id=IOAI7IFMLDABT2H7RSCR3TQH7QA7PVTM4FSSJ7KLEI5ELHR337PA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:39:58.302482007 (60h4m8.025045047s) [noise=true , address=77.90.31.13:4627 , piece_id=VR7ZYG7CAY26LNQB4KXSLZ67AQ3YCNYZSDSPQUAJH4VAUUWMSB4Q, seg_pos={0 0}, upload=57629]: Done uploading piece. err: <nil> Jul 26 23:40:23.972358768 (60h4m33.694921808s) [noise=true , address=104.36.82.3:4767 , piece_id=K3J4WV5GD2MYO2ZZ6MKDTOGM3PBKJENJ2EGHYDFI55BFAZW5V7IQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:42548->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972425154 (60h4m33.694988205s) [noise=true , address=104.36.82.7:4912 , piece_id=HWKVVPMIBSYGOVFRDXZS2NCSIWQZH4LJUJ3V3SDME2QDOR2TQM6A, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:39050->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972494999 (60h4m33.695058059s) [noise=true , address=104.36.82.5:4840 , piece_id=X35OIBIFJVBKWZ2TS4MBCA6J4GKYICZTMH5ZOEODF6IDONHYIAAA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:55986->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972672428 (60h4m33.695235478s) [noise=true , address=104.36.82.2:4732 , piece_id=NHZWL4O44AKJ2GIZ6KDO5KM66JIM42TZTXZR3VWJKIHFLWQJ4GCQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972730348 (60h4m33.695293388s) [noise=true , address=104.36.82.14:4223 , piece_id=DEWIE4BGTOCEFQ3MRMQTPL34K7MPH4QNO3OYKINH4DXSNJJR7CIA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:53096->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972784672 (60h4m33.695347732s) [noise=true , address=104.36.82.12:4150 , piece_id=OID3AOLTTEGHC7FI4ABETMTI5TIM7IALBJHE3IX2DFBA756LAMQQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:45708->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972819510 (60h4m33.69538257s) [noise=true , address=104.36.82.16:4295 , piece_id=T2Z3M7JYTMCT3X2AHFDMHCWBV23T33BKOUMUJY4S4THMBGYQDROQ, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:56532->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972877450 (60h4m33.69544049s) [noise=true , address=104.36.82.9:4007 , piece_id=N2M5H3LFJWZKMGKRCHLANERL2MBT7LERMRECWVKBPEQB2GCVGPIA, seg_pos={0 0}, upload=57629]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:58560->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.972922356 (60h4m33.695485406s) [seg_pos={0 0}, upload=57629]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:40:24.202943213 (60h4m33.925506313s) [upload=57629]: Done uploading segments... Jul 26 23:40:24.202967600 (60h4m33.92553064s) [upload=57629]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.232663224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.829142391s FAIL (2025-07-26 23:40:12 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.829165185s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X...] duration: 42.683228493s FAIL (2025-07-26 23:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.683241828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.22957946s FAIL (2025-07-26 23:39:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== upload=57617]: Begin upload segment... Jul 26 23:39:00.563533224 (60h3m10.286096264s) [seg_pos={0 0}, upload=57617]: Done begin upload segment. Jul 26 23:39:00.563555096 (60h3m10.286118136s) [noise=true , address=79.127.201.215:4358 , piece_id=OPORJ3WXI6TSU2SMTWLJUT46LH5PGU2VIUWDBHMN7D3PIVBUIHDQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.563617165 (60h3m10.286180205s) [noise=true , address=45.91.116.14:4194 , piece_id=HNEDBA7KJKSUGQQGJZ7MPAD6X5U5REJFHPOI5ULX3SBXPPQFXHSQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.563949611 (60h3m10.286512651s) [noise=true , address=104.36.82.14:4241 , piece_id=H6GYPIUJ7YIBWR5L627PHSZGW5FFKQOO2IHFIXTLTRTAY63YEDHA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564133212 (60h3m10.286696252s) [noise=true , address=64.49.13.138:4483 , piece_id=OH3CTMXK3DVWLN7WNAQ5FLP2K3JH6OIPATAK2Y6JN44XSMAZ3AAQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564296865 (60h3m10.286859906s) [noise=true , address=79.127.205.234:4606 , piece_id=AAHNXSTSO6H4VAC4TBVZGZXGNFZIY6IBR6WT6HYOJAXUMMQYNUGA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564548687 (60h3m10.287111727s) [noise=true , address=79.127.205.246:4262 , piece_id=RL6WH366VAZ24OVWMRATCUMSCNLJCR6HNASEEKOWCCBP6VTJ7MXQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564577853 (60h3m10.287140893s) [noise=true , address=45.91.116.9:4277 , piece_id=4LL6JDY53XTTHQJLQP325DP7KYF6ISYVDJTEWU6OFCP7MAPYM5VQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564806290 (60h3m10.28736934s) [noise=true , address=104.36.82.54:4173 , piece_id=D5AHPXYPRAO6IDETDNBPSARU67DS3UBJLV7X532MCESUSEEZSGUA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.564888347 (60h3m10.287451387s) [noise=true , address=79.127.201.215:4355 , piece_id=3PWIAFHOLA6OESPI6AKJW35FLVXI6KU2YIOW55E5JEZIIJ2GNK5Q, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565031360 (60h3m10.2875944s) [noise=true , address=104.36.82.51:4060 , piece_id=K2NH4DHMZAVUULNGZ36HYG43QQEPRJ5SR7QRERZ37VPJVP2HIOFA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565139717 (60h3m10.287702758s) [noise=true , address=104.36.82.46:4508 , piece_id=KBXR3IQDB2TDYEAXBJLXHUHCIPUDLPARSXCND75HMMEGF4LYNBAQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565264657 (60h3m10.287827697s) [noise=true , address=104.36.82.53:4128 , piece_id=4B6DJQ4ZD56EFO2HYEJOQQRAHJZMN2NVXFGYOBJ5RLQITVAF4WDA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565440523 (60h3m10.288003553s) [noise=true , address=77.90.31.10:4540 , piece_id=3YSXDHGJP44FOOWMG3LJLPTOYJLDCISXREIRLG53GEU32GWSZ55A, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565480280 (60h3m10.28804332s) [noise=true , address=79.127.205.245:4244 , piece_id=FKXN2W6NZIQ24Q5WG3XBCSDRAP77Y73ZJDNGHE3IANEJB4SYYXGA, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565681094 (60h3m10.288244134s) [noise=true , address=45.91.116.9:4273 , piece_id=UX7QDOCZR3THS7AZOMQAYQSYFKJLQGYRIGIMWLDSWTZAKSDQALTQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.565838465 (60h3m10.288401505s) [noise=true , address=77.90.31.12:4599 , piece_id=QJZOQ4YNLEP4Y5ILOFSNPDKG6NKOPEX3MRF4UU4I7GBGYQ2RKLCQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.566095657 (60h3m10.288658697s) [noise=true , address=77.90.31.11:4559 , piece_id=WN3CHPURJOHLSHQGCSUXQGRNDKGAI7EKS6GYV3L6S2U6WAEM2PIQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:00.569345735 (60h3m10.291908785s) [noise=true , address=156.146.43.225:4449 , piece_id=LUBMQ6EFDU5775O7HUT3YJZT6FRS5XI7S47ITF6DVEO6PNMSCQWQ, seg_pos={0 0}, upload=57617]: Uploading piece... Jul 26 23:39:01.590872054 (60h3m11.313435094s) [noise=true , address=156.146.43.225:4449 , piece_id=LUBMQ6EFDU5775O7HUT3YJZT6FRS5XI7S47ITF6DVEO6PNMSCQWQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:39:01.664029455 (60h3m11.386592495s) [upload=57617]: Next returned nil segment Jul 26 23:39:01.664046087 (60h3m11.386609127s) [upload=57617]: Waiting for error group managing segments... Jul 26 23:39:01.942806661 (60h3m11.665369701s) [noise=true , address=79.127.205.234:4606 , piece_id=AAHNXSTSO6H4VAC4TBVZGZXGNFZIY6IBR6WT6HYOJAXUMMQYNUGA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:01.950304697 (60h3m11.672867737s) [noise=true , address=79.127.205.246:4262 , piece_id=RL6WH366VAZ24OVWMRATCUMSCNLJCR6HNASEEKOWCCBP6VTJ7MXQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:01.966901802 (60h3m11.689464842s) [noise=true , address=79.127.205.245:4244 , piece_id=FKXN2W6NZIQ24Q5WG3XBCSDRAP77Y73ZJDNGHE3IANEJB4SYYXGA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:02.531205896 (60h3m12.253768936s) [noise=true , address=45.91.116.9:4277 , piece_id=4LL6JDY53XTTHQJLQP325DP7KYF6ISYVDJTEWU6OFCP7MAPYM5VQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:02.566336792 (60h3m12.288899832s) [noise=true , address=77.90.31.12:4599 , piece_id=QJZOQ4YNLEP4Y5ILOFSNPDKG6NKOPEX3MRF4UU4I7GBGYQ2RKLCQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:03.693300014 (60h3m13.415863055s) [noise=true , address=45.91.116.9:4273 , piece_id=UX7QDOCZR3THS7AZOMQAYQSYFKJLQGYRIGIMWLDSWTZAKSDQALTQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:07.489158787 (60h3m17.211721857s) [noise=true , address=45.91.116.14:4194 , piece_id=HNEDBA7KJKSUGQQGJZ7MPAD6X5U5REJFHPOI5ULX3SBXPPQFXHSQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:11.678653795 (60h3m21.401216835s) [noise=true , address=64.49.13.138:4483 , piece_id=OH3CTMXK3DVWLN7WNAQ5FLP2K3JH6OIPATAK2Y6JN44XSMAZ3AAQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:13.481650041 (60h3m23.204213081s) [noise=true , address=79.127.201.215:4355 , piece_id=3PWIAFHOLA6OESPI6AKJW35FLVXI6KU2YIOW55E5JEZIIJ2GNK5Q, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:32.586856572 (60h3m42.309419602s) [noise=true , address=79.127.201.215:4358 , piece_id=OPORJ3WXI6TSU2SMTWLJUT46LH5PGU2VIUWDBHMN7D3PIVBUIHDQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: <nil> Jul 26 23:39:59.472484714 (60h4m9.195047754s) [noise=true , address=77.90.31.10:4540 , piece_id=3YSXDHGJP44FOOWMG3LJLPTOYJLDCISXREIRLG53GEU32GWSZ55A, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:39:59.472597730 (60h4m9.19516076s) [noise=true , address=104.36.82.54:4173 , piece_id=D5AHPXYPRAO6IDETDNBPSARU67DS3UBJLV7X532MCESUSEEZSGUA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472687112 (60h4m9.195250152s) [noise=true , address=104.36.82.46:4508 , piece_id=KBXR3IQDB2TDYEAXBJLXHUHCIPUDLPARSXCND75HMMEGF4LYNBAQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:39:59.472742578 (60h4m9.195305608s) [noise=true , address=104.36.82.53:4128 , piece_id=4B6DJQ4ZD56EFO2HYEJOQQRAHJZMN2NVXFGYOBJ5RLQITVAF4WDA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: rpc: dial tcp 104.36.82.53:4128: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472831308 (60h4m9.195394348s) [noise=true , address=104.36.82.14:4241 , piece_id=H6GYPIUJ7YIBWR5L627PHSZGW5FFKQOO2IHFIXTLTRTAY63YEDHA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472872005 (60h4m9.195435045s) [noise=true , address=104.36.82.51:4060 , piece_id=K2NH4DHMZAVUULNGZ36HYG43QQEPRJ5SR7QRERZ37VPJVP2HIOFA, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:19VJKnTAges9YKYdUVoz3tb3sFU1bawxpaJ2rsBd2wmLo1zK4X, address:104.36.82.51:4060): protocol: write tcp 188.166.225.170:50200->104.36.82.51:4060: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:59.472895179 (60h4m9.19545822s) [noise=true , address=77.90.31.11:4559 , piece_id=WN3CHPURJOHLSHQGCSUXQGRNDKGAI7EKS6GYV3L6S2U6WAEM2PIQ, seg_pos={0 0}, upload=57617]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: read tcp 188.166.225.170:53148->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:39:59.472914677 (60h4m9.195477717s) [seg_pos={0 0}, upload=57617]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:39:59.700511699 (60h4m9.423074739s) [upload=57617]: Done uploading segments... Jul 26 23:39:59.700537529 (60h4m9.423100569s) [upload=57617]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.229592836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237310537s FAIL (2025-07-26 23:39:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== d enough pieces (needed at least 13 but got 4) Jul 26 23:39:24.147360527 (60h3m33.869923567s) [upload=57624]: Starting upload Jul 26 23:39:24.147422626 (60h3m33.869985666s) [upload=57624]: Done starting upload Jul 26 23:39:24.147510373 (60h3m33.870073414s) [upload=57624]: Uploading segments... Jul 26 23:39:24.147595657 (60h3m33.870158697s) [upload=57624]: Got next segment. Inline: false Jul 26 23:39:24.147621466 (60h3m33.870184506s) [upload=57624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:24.147625765 (60h3m33.870188805s) [upload=57624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:24.476822648 (60h3m34.199385688s) [seg_pos={0 0}, upload=57624]: Begin upload segment... Jul 26 23:39:24.476871762 (60h3m34.199434802s) [seg_pos={0 0}, upload=57624]: Done begin upload segment. Jul 26 23:39:24.476894746 (60h3m34.199457776s) [noise=true , address=104.36.82.4:4804 , piece_id=GFBUBQ2WZVH7DKBMJ26DG4BVFGFZXKBKNLTQPXFZE3E32EW4IXIA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.476910366 (60h3m34.199473406s) [noise=true , address=104.36.82.13:4187 , piece_id=S7HJZC2V3TOOUNVBRHUCT47URKQ6EA3XDBDFNWNPWKS6MNJI5MGA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477231360 (60h3m34.19979439s) [noise=true , address=104.36.82.3:4767 , piece_id=CJDCRY3JYY7XV6VBSRVJCSAWUIM7KAD6OAVYFHVDTZWWSPOTLR5A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477234686 (60h3m34.199797726s) [noise=true , address=45.91.116.4:4938 , piece_id=PR5XK5SF7WIC3QGCOFFHIDG5CYNF3TWBHUPYQ46VZK7HY6XGLZTA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477485145 (60h3m34.200048185s) [noise=true , address=45.91.116.8:4047 , piece_id=VRIUESCP4JHUXIFA73F244RPKOPRZC56JENBLKZBFNONJHN7EDFQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477491468 (60h3m34.200054508s) [noise=true , address=45.91.116.10:4083 , piece_id=P3ID2XHSJHUSIGL3HSKCLXPQQVGW6K2UCEDAV5XKNAEY5G67Z52A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477738511 (60h3m34.200301541s) [noise=true , address=104.36.82.5:4839 , piece_id=W7MD6NGGBGKJWPLQTZXF2EX233K54WK4MI5DMGALW6QTED4XCHJA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477745805 (60h3m34.200308835s) [noise=true , address=77.90.31.14:4661 , piece_id=MSRPMZQZXRK2FC5U6ISJPGMMGUOASHJYTLRJYUGYQ4RB4F5HXEAQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.477971576 (60h3m34.200534616s) [noise=true , address=45.91.116.7:4011 , piece_id=RDUDNWPDME2ASGGGNJSDAX3I3QRRXIG27PAOVB3M5I4MM2DVJRLQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478067841 (60h3m34.200630881s) [noise=true , address=104.36.82.12:4150 , piece_id=F4VXQSQCPSOND7BNAI3N6Y3CT7SBACEHUHS5WVM3IQ4L24RTBL4A, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478227356 (60h3m34.200790396s) [noise=true , address=45.91.116.6:4973 , piece_id=VENHMAXJFIREXN5EYT5N6VE3VL5HCRNRPZSX3CNIHU5PHVIKRTMA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478305595 (60h3m34.200868625s) [noise=true , address=104.36.82.15:4259 , piece_id=5OSOZR6RWHAEWIU2ECCQMXZV2IBJBFQ5PRZAEEH32R5C7X55U3PA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478525386 (60h3m34.201088426s) [noise=true , address=77.90.31.19:4841 , piece_id=RQELPXBAA7OOG7VDJZYOGWB5VCLPNL6Y3R5CHM6VJ3DITTN6Q6ZQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.478747010 (60h3m34.20131004s) [noise=true , address=45.91.116.5:4298 , piece_id=SHWJLIZ4NYKAYFDYXB2TJCFUSAEPW3DSJQSZNBBRWQA7HYXGYLCQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.479153227 (60h3m34.201716267s) [noise=true , address=104.36.82.14:4223 , piece_id=RH4OJUFBJWBMHA7HQ6WBJJ4MX7G6AI7RUNVL4MHPUV5RHY55YSRA, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:24.479525179 (60h3m34.202088219s) [noise=true , address=104.36.82.6:4873 , piece_id=FBWXVEFQ65ZUZV636WDYOHIXKTHQBK7LBGT6735GQN56XZ4UM2WQ, seg_pos={0 0}, upload=57624]: Uploading piece... Jul 26 23:39:26.192582066 (60h3m35.915145106s) [upload=57624]: Next returned nil segment Jul 26 23:39:26.192598226 (60h3m35.915161266s) [upload=57624]: Waiting for error group managing segments... Jul 26 23:39:26.488939757 (60h3m36.211502797s) [noise=true , address=45.91.116.4:4938 , piece_id=PR5XK5SF7WIC3QGCOFFHIDG5CYNF3TWBHUPYQ46VZK7HY6XGLZTA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.502341068 (60h3m36.224904098s) [noise=true , address=45.91.116.5:4298 , piece_id=SHWJLIZ4NYKAYFDYXB2TJCFUSAEPW3DSJQSZNBBRWQA7HYXGYLCQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.510343499 (60h3m36.232906539s) [noise=true , address=45.91.116.7:4011 , piece_id=RDUDNWPDME2ASGGGNJSDAX3I3QRRXIG27PAOVB3M5I4MM2DVJRLQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.589409295 (60h3m36.311972335s) [noise=true , address=77.90.31.14:4661 , piece_id=MSRPMZQZXRK2FC5U6ISJPGMMGUOASHJYTLRJYUGYQ4RB4F5HXEAQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.589853746 (60h3m36.312416796s) [noise=true , address=45.91.116.6:4973 , piece_id=VENHMAXJFIREXN5EYT5N6VE3VL5HCRNRPZSX3CNIHU5PHVIKRTMA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.591676964 (60h3m36.314240004s) [noise=true , address=45.91.116.8:4047 , piece_id=VRIUESCP4JHUXIFA73F244RPKOPRZC56JENBLKZBFNONJHN7EDFQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:26.655348387 (60h3m36.377911427s) [noise=true , address=45.91.116.10:4083 , piece_id=P3ID2XHSJHUSIGL3HSKCLXPQQVGW6K2UCEDAV5XKNAEY5G67Z52A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:28.117305011 (60h3m37.839868051s) [noise=true , address=77.90.31.19:4841 , piece_id=RQELPXBAA7OOG7VDJZYOGWB5VCLPNL6Y3R5CHM6VJ3DITTN6Q6ZQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: <nil> Jul 26 23:39:53.503295136 (60h4m3.225858166s) [noise=true , address=104.36.82.15:4259 , piece_id=5OSOZR6RWHAEWIU2ECCQMXZV2IBJBFQ5PRZAEEH32R5C7X55U3PA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503348258 (60h4m3.225911288s) [noise=true , address=104.36.82.5:4839 , piece_id=W7MD6NGGBGKJWPLQTZXF2EX233K54WK4MI5DMGALW6QTED4XCHJA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:58420->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503452648 (60h4m3.226015688s) [noise=true , address=104.36.82.3:4767 , piece_id=CJDCRY3JYY7XV6VBSRVJCSAWUIM7KAD6OAVYFHVDTZWWSPOTLR5A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503485220 (60h4m3.22604826s) [noise=true , address=104.36.82.4:4804 , piece_id=GFBUBQ2WZVH7DKBMJ26DG4BVFGFZXKBKNLTQPXFZE3E32EW4IXIA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:34774->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503501671 (60h4m3.226064711s) [noise=true , address=104.36.82.12:4150 , piece_id=F4VXQSQCPSOND7BNAI3N6Y3CT7SBACEHUHS5WVM3IQ4L24RTBL4A, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503527280 (60h4m3.22609032s) [noise=true , address=104.36.82.13:4187 , piece_id=S7HJZC2V3TOOUNVBRHUCT47URKQ6EA3XDBDFNWNPWKS6MNJI5MGA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503534935 (60h4m3.226097975s) [noise=true , address=104.36.82.14:4223 , piece_id=RH4OJUFBJWBMHA7HQ6WBJJ4MX7G6AI7RUNVL4MHPUV5RHY55YSRA, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503566025 (60h4m3.226129065s) [noise=true , address=104.36.82.6:4873 , piece_id=FBWXVEFQ65ZUZV636WDYOHIXKTHQBK7LBGT6735GQN56XZ4UM2WQ, seg_pos={0 0}, upload=57624]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:53.503593728 (60h4m3.226156768s) [seg_pos={0 0}, upload=57624]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:39:53.738260936 (60h4m3.460823976s) [upload=57624]: Done uploading segments... Jul 26 23:39:53.738292727 (60h4m3.460855847s) [upload=57624]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.237340895s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.] duration: 27.7280688s FAIL (2025-07-26 23:39:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.728088998s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235054871s FAIL (2025-07-26 23:39:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23507538s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X..] duration: 46.905355123s FAIL (2025-07-26 23:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.905368018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.290008255s FAIL (2025-07-26 23:39:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.290019787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [X.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232243318s FAIL (2025-07-26 23:38:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== .82.6:4899 , piece_id=GHNDWYDAJ5ULHBZS5W566CUGVAQNAJYXSXNJ4Y7OKVUXQMJ3O2MA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116496262 (60h2m9.839059312s) [noise=true , address=77.90.31.18:4815 , piece_id=PPQFNBEH5J6R6YS2YHY63T4O7LJH7QPUFHCX6BGV5ZLUSZE6GRMQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116743264 (60h2m9.839306304s) [noise=true , address=77.90.31.21:4905 , piece_id=HHUOZXJUW6HHRB6PDZ6Z62K27DY27LB644UO4W2K5PI2HSJMH3NQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116901738 (60h2m9.839464778s) [noise=true , address=79.127.205.234:4613 , piece_id=EPLKZP6ZFV6SMXWQVQ5EAS37XBZZWI35ZXXHC77OUJPVZJ5T4MSA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.116993653 (60h2m9.839556683s) [noise=true , address=79.127.219.38:4015 , piece_id=YIFJC6QCHKBOA7WJHJVEHNABNB55NSDTFAFWAMEZR7LGDRXPGFRQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117239413 (60h2m9.839802453s) [noise=true , address=79.127.205.235:4667 , piece_id=JZR6IERCRWL65D57LGXOLCK7FABAZK2CC3426PLHZVPO55EAZZTA, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117485364 (60h2m9.840048404s) [noise=true , address=104.36.82.55:4224 , piece_id=ATSAO7QMB43NHCL4IGW2APVOACK7FHU4LMQ4OIELBRNBWXEG3VHQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117562742 (60h2m9.840125782s) [noise=true , address=104.36.82.15:4273 , piece_id=A4M2WE67BHCHFYK7VYIICVVCHOMQBJAUMFQVP3U3AFOBC6EVGSSQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117702249 (60h2m9.840265289s) [noise=true , address=104.36.82.56:4261 , piece_id=2A34GGYXGU3WH3CDZARWEBLUQUGDABNSZXYGJG4BNMUONCYFYHPQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.117934844 (60h2m9.840497884s) [noise=true , address=79.127.201.215:4345 , piece_id=XCE3GOM2XSR5TE2F65IT7K7KVKC354IZ6WMSLCI6SLHWWT7BDDYQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118055555 (60h2m9.840618595s) [noise=true , address=79.127.220.97:4361 , piece_id=KZHYYS7FHM4ZAUBLXKL3D7EYNXJ3IUI7NTIRAP6JVI3ASTLMF72Q, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118364246 (60h2m9.840927286s) [noise=true , address=77.90.31.13:4638 , piece_id=OLKFQDFUBERTMW6PIRJJA4KO5R43IZ3RGLDQYCDW4TLLBOJPAWTQ, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:00.118644291 (60h2m9.841207331s) [noise=true , address=64.49.13.138:4482 , piece_id=YJDJYD5GFH7Q27Z6GWX3SBSW2VMGXI5M4F3BWYU7EW3PCV322X7A, seg_pos={0 0}, upload=57602]: Uploading piece... Jul 26 23:38:01.252588591 (60h2m10.975151631s) [upload=57602]: Next returned nil segment Jul 26 23:38:01.252606215 (60h2m10.975169255s) [upload=57602]: Waiting for error group managing segments... Jul 26 23:38:01.539602033 (60h2m11.262165063s) [noise=true , address=79.127.205.234:4613 , piece_id=EPLKZP6ZFV6SMXWQVQ5EAS37XBZZWI35ZXXHC77OUJPVZJ5T4MSA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:01.564041003 (60h2m11.286604043s) [noise=true , address=79.127.205.235:4667 , piece_id=JZR6IERCRWL65D57LGXOLCK7FABAZK2CC3426PLHZVPO55EAZZTA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.006438754 (60h2m11.729001794s) [noise=true , address=79.127.219.38:4015 , piece_id=YIFJC6QCHKBOA7WJHJVEHNABNB55NSDTFAFWAMEZR7LGDRXPGFRQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.118334845 (60h2m11.840897915s) [noise=true , address=77.90.31.18:4815 , piece_id=PPQFNBEH5J6R6YS2YHY63T4O7LJH7QPUFHCX6BGV5ZLUSZE6GRMQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:02.806564256 (60h2m12.529127296s) [noise=true , address=64.49.13.138:4482 , piece_id=YJDJYD5GFH7Q27Z6GWX3SBSW2VMGXI5M4F3BWYU7EW3PCV322X7A, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:38:05.670956897 (60h2m15.393519937s) [noise=true , address=79.127.220.97:4361 , piece_id=KZHYYS7FHM4ZAUBLXKL3D7EYNXJ3IUI7NTIRAP6JVI3ASTLMF72Q, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:09.006182463 (60h2m18.728745503s) [noise=true , address=79.127.201.215:4345 , piece_id=XCE3GOM2XSR5TE2F65IT7K7KVKC354IZ6WMSLCI6SLHWWT7BDDYQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: <nil> Jul 26 23:38:59.016355526 (60h3m8.738918606s) [noise=true , address=104.36.82.56:4261 , piece_id=2A34GGYXGU3WH3CDZARWEBLUQUGDABNSZXYGJG4BNMUONCYFYHPQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: write tcp 188.166.225.170:51566->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016434367 (60h3m8.738997407s) [noise=true , address=104.36.82.12:4183 , piece_id=WVC63ZGZVLBCVSQJHQKSP3HK3QW5MNYRJ3K54UWHZIRYVK3WS6FQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: write tcp 188.166.225.170:57746->104.36.82.12:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016476327 (60h3m8.739039367s) [noise=true , address=104.36.82.47:4527 , piece_id=ULI54HPWTC4WR3INH4OE5TSI5M3SKK5ZAF6MIOEWL6QJQLGQSQAQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 188.166.225.170:38294->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016495514 (60h3m8.739058544s) [noise=true , address=77.90.31.17:4801 , piece_id=7MNLNQK3ZD6SDGZKJFN4FAQ6V4FNVCKWAVJ76JCO47Z7VRL7QRPA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: write tcp 188.166.225.170:43164->77.90.31.17:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016510403 (60h3m8.739073443s) [noise=true , address=104.36.82.22:4521 , piece_id=QMNMNIELBPAA6NVS4N2J6V7QWO7HVO4YKOJS57NG5EE2T2RROQMA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1zWiwpaV58h8YKsAgw24fUWHBBYYyqF4UX6GJV316Uxo9Y7gt1, address:104.36.82.22:4521): protocol: write tcp 188.166.225.170:42360->104.36.82.22:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016542724 (60h3m8.739105764s) [noise=true , address=104.36.82.6:4899 , piece_id=GHNDWYDAJ5ULHBZS5W566CUGVAQNAJYXSXNJ4Y7OKVUXQMJ3O2MA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 188.166.225.170:38902->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016548655 (60h3m8.739111695s) [noise=true , address=77.90.31.13:4638 , piece_id=OLKFQDFUBERTMW6PIRJJA4KO5R43IZ3RGLDQYCDW4TLLBOJPAWTQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:38:59.016553785 (60h3m8.739116825s) [noise=true , address=104.36.82.55:4224 , piece_id=ATSAO7QMB43NHCL4IGW2APVOACK7FHU4LMQ4OIELBRNBWXEG3VHQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 188.166.225.170:54644->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016571960 (60h3m8.739135s) [noise=true , address=104.36.82.15:4273 , piece_id=A4M2WE67BHCHFYK7VYIICVVCHOMQBJAUMFQVP3U3AFOBC6EVGSSQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1cgTYjzrDTV5a4aohnpZvGPRweYkRv2dyKhQw46nQwMtAK3Kuv, address:104.36.82.15:4273): protocol: write tcp 188.166.225.170:57148->104.36.82.15:4273: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016605394 (60h3m8.739168434s) [noise=true , address=104.36.82.46:4481 , piece_id=2IBGLYHQR33S7VQ57EUVIM6TN26AWOPCOWYOVM2YU2YVRQHQZGBA, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:57564->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016618650 (60h3m8.7391817s) [noise=true , address=77.90.31.21:4905 , piece_id=HHUOZXJUW6HHRB6PDZ6Z62K27DY27LB644UO4W2K5PI2HSJMH3NQ, seg_pos={0 0}, upload=57602]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 188.166.225.170:44248->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:59.016645120 (60h3m8.739208161s) [seg_pos={0 0}, upload=57602]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:38:59.246727247 (60h3m8.969290327s) [upload=57602]: Done uploading segments... Jul 26 23:38:59.246749859 (60h3m8.969312929s) [upload=57602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.232255691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.943785459s FAIL (2025-07-26 23:38:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.943810848s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....................XXX.X..X.....XXXXX...XXX.X.....X.X..X..X] duration: 30.001112162s FAIL (2025-07-26 23:38:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001134443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [.........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234209214s FAIL (2025-07-26 23:38:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== m24.391763459s) [noise=true , address=104.36.82.5:4840 , piece_id=R3KN7BXF5WUZBZV7JILT6BOXE32WNHZSO6HPIT5KVOIQOZKTWAVQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.669239934 (60h2m24.391802974s) [noise=true , address=104.36.82.14:4221 , piece_id=GXRK5RM4WE2KWQXLM25WNXSU7U7X5WJVFTGUTJ6XD5CPPUYT4UXQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.669507606 (60h2m24.392070646s) [noise=true , address=104.36.82.10:4078 , piece_id=CVRHX3UNI7DEMBXQ3YPTV23EMWEVWP5YM3QDDCDDVX3Q5YY57H6Q, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670240638 (60h2m24.392803678s) [noise=true , address=104.36.82.6:4875 , piece_id=7GEBDKM3ZNBO3NJBVQW3PI2VDRURMKM75UP5GJ3X6HZPIZJY3YMA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670374474 (60h2m24.392937514s) [noise=true , address=104.36.82.10:4077 , piece_id=57APRH5AKKCJFT2UL644YESH7CAVPXGR4BDUVRAZ2LTZ7FMOTGQA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.670757387 (60h2m24.393320427s) [noise=true , address=77.90.31.16:4734 , piece_id=AZJ3LBGUQDAJCAQ7YMHJTZCMPADTTWZGNVA2OOJSTIACGHMBO2YA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.671048555 (60h2m24.393611595s) [noise=true , address=104.36.82.13:4187 , piece_id=M42HABRDVKHENZJVCR6P4XQIS4P2HZ23G34AZC2GDFESZLISKJ2A, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.671957163 (60h2m24.394520203s) [noise=true , address=104.36.82.3:4765 , piece_id=GYHURFAGQLPNAGKYIHDBOJ5JMEMFZULYS763DPDCGTH3KIPV3BIQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.672284149 (60h2m24.394847189s) [noise=true , address=104.36.82.13:4185 , piece_id=JIILB7TYWN625HZC3DSVQIWKNTXA6ZKSYBXIZV4FKZ75IFU4DT2Q, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.673948042 (60h2m24.396511082s) [noise=true , address=104.36.82.16:4293 , piece_id=OVSA675FVUNB3USAQ4DEZMUH6IQD6C5PQJQCYBPUXMFEQLZBAOVA, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674232486 (60h2m24.396795526s) [noise=true , address=104.36.82.7:4912 , piece_id=GAG2M536ZN3NMECWQWDZI2WDXMUGE7Q2IMJ3XZORC7SSLV3FRBEQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674534263 (60h2m24.397097303s) [noise=true , address=104.36.82.8:4972 , piece_id=COM2PORJRNEIZBRMFMYJXKEHDXMXDBD7RLSVRFHYV4BXUBO3FJIQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.674820932 (60h2m24.397383972s) [noise=true , address=45.91.116.10:4082 , piece_id=MF27QJDYHV22G44G43RD2T664Y43P3XM6BH26JEPKGYDMW4W3JCQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:14.675080538 (60h2m24.397643578s) [noise=true , address=45.91.116.8:4047 , piece_id=GHHFYWMOBEMJPAFVDI2ZRTQC2RVYFNYAP4CMPYEXRYTUXAIJVADQ, seg_pos={0 0}, upload=57606]: Uploading piece... Jul 26 23:38:16.420628258 (60h2m26.143191299s) [upload=57606]: Next returned nil segment Jul 26 23:38:16.420645290 (60h2m26.14320832s) [upload=57606]: Waiting for error group managing segments... Jul 26 23:38:16.697463388 (60h2m26.420026438s) [noise=true , address=45.91.116.7:4009 , piece_id=J5NUILU5V7DOOYU6DZTIKGU64F64QITWMFLRN76LBUACWFJZRLMQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.708920047 (60h2m26.431483087s) [noise=true , address=77.90.31.16:4734 , piece_id=AZJ3LBGUQDAJCAQ7YMHJTZCMPADTTWZGNVA2OOJSTIACGHMBO2YA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.846707878 (60h2m26.569270918s) [noise=true , address=45.91.116.8:4047 , piece_id=GHHFYWMOBEMJPAFVDI2ZRTQC2RVYFNYAP4CMPYEXRYTUXAIJVADQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:16.871264612 (60h2m26.593827642s) [noise=true , address=45.91.116.10:4082 , piece_id=MF27QJDYHV22G44G43RD2T664Y43P3XM6BH26JEPKGYDMW4W3JCQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: <nil> Jul 26 23:38:43.795759508 (60h2m53.518322548s) [noise=true , address=104.36.82.10:4078 , piece_id=CVRHX3UNI7DEMBXQ3YPTV23EMWEVWP5YM3QDDCDDVX3Q5YY57H6Q, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:43776->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795812639 (60h2m53.518375679s) [noise=true , address=104.36.82.10:4079 , piece_id=7Z6D4HRRYMAQHE3JSBA4ATDUHYJ3RPC7OAZDSILSRAA7T3U3RUOQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:36432->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795846314 (60h2m53.518409354s) [noise=true , address=104.36.82.6:4875 , piece_id=7GEBDKM3ZNBO3NJBVQW3PI2VDRURMKM75UP5GJ3X6HZPIZJY3YMA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:59630->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795878636 (60h2m53.518441676s) [noise=true , address=104.36.82.3:4765 , piece_id=GYHURFAGQLPNAGKYIHDBOJ5JMEMFZULYS763DPDCGTH3KIPV3BIQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:42150->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795899916 (60h2m53.518462956s) [noise=true , address=104.36.82.14:4221 , piece_id=GXRK5RM4WE2KWQXLM25WNXSU7U7X5WJVFTGUTJ6XD5CPPUYT4UXQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:33240->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.795977294 (60h2m53.518540324s) [noise=true , address=104.36.82.16:4293 , piece_id=OVSA675FVUNB3USAQ4DEZMUH6IQD6C5PQJQCYBPUXMFEQLZBAOVA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:58082->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796015548 (60h2m53.518578588s) [noise=true , address=104.36.82.8:4972 , piece_id=COM2PORJRNEIZBRMFMYJXKEHDXMXDBD7RLSVRFHYV4BXUBO3FJIQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:45332->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796051086 (60h2m53.518614126s) [noise=true , address=104.36.82.5:4840 , piece_id=R3KN7BXF5WUZBZV7JILT6BOXE32WNHZSO6HPIT5KVOIQOZKTWAVQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796087886 (60h2m53.518650926s) [noise=true , address=104.36.82.7:4912 , piece_id=GAG2M536ZN3NMECWQWDZI2WDXMUGE7Q2IMJ3XZORC7SSLV3FRBEQ, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:55634->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796125979 (60h2m53.518689019s) [noise=true , address=104.36.82.13:4187 , piece_id=M42HABRDVKHENZJVCR6P4XQIS4P2HZ23G34AZC2GDFESZLISKJ2A, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 188.166.225.170:34478->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796145877 (60h2m53.518708917s) [noise=true , address=104.36.82.10:4077 , piece_id=57APRH5AKKCJFT2UL644YESH7CAVPXGR4BDUVRAZ2LTZ7FMOTGQA, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:45686->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796149734 (60h2m53.518712774s) [noise=true , address=104.36.82.13:4185 , piece_id=JIILB7TYWN625HZC3DSVQIWKNTXA6ZKSYBXIZV4FKZ75IFU4DT2Q, seg_pos={0 0}, upload=57606]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:49934->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:43.796189531 (60h2m53.518752571s) [seg_pos={0 0}, upload=57606]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:38:44.028591695 (60h2m53.751154735s) [upload=57606]: Done uploading segments... Jul 26 23:38:44.028617073 (60h2m53.751180113s) [upload=57606]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.234231737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.....X.] duration: 49.881669679s FAIL (2025-07-26 23:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.881690869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.230648197s FAIL (2025-07-26 23:38:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.230658908s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.838478015s FAIL (2025-07-26 23:38:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.838508092s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234574129s FAIL (2025-07-26 23:38:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.234600701s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233703339s FAIL (2025-07-26 23:37:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== :59.680062859 (60h1m9.402625899s) [noise=true , address=79.127.219.38:4015 , piece_id=JPSZ3NOQAAYJRKRGPWK3U7HUHD2SU7EFCNACKYE5K7ZE7WOS7DNA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680144155 (60h1m9.402707195s) [noise=true , address=79.127.205.235:4677 , piece_id=65OUDRPAZWTNUPPWOAKM6J73GZU7EL3K6UTGEDJ2M24XIGZ22FYA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680481611 (60h1m9.403044651s) [noise=true , address=77.90.31.18:4821 , piece_id=DJXMXCRUBMOB4SQACXOX6376MUA4CQRWEOJAK3N2E6DO6PFV3IPQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680610417 (60h1m9.403173457s) [noise=true , address=104.36.82.21:4493 , piece_id=Q52S6O65QUB3LSC66EPGY62ESVGB7RQZY2W7I6WUZK2Z4ILFDOQA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680784491 (60h1m9.403347531s) [noise=true , address=79.127.205.233:4639 , piece_id=MYHLASW5TLXLY6SNYA4GGP4WLMZJ6QEBPGZU3OVHO5GT2WWU4M4A, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.680879031 (60h1m9.403442071s) [noise=true , address=104.36.82.47:4547 , piece_id=NFC2PLVBXUQZTEZZXRP7WMSIOPGJHWFDGHL35Q7NWQ74ZO2RBAVA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681046291 (60h1m9.403609331s) [noise=true , address=79.127.205.246:4274 , piece_id=YDQ4WQATQITXAGG4BXS6YSDR2BQRIZN3DYH2MFJJVTITQIYM54IQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681183113 (60h1m9.403746153s) [noise=true , address=77.90.31.16:4748 , piece_id=QMC2ZIYYHWQYGJFZGWGU4HE4CGT7DGCCWMJNF7JJKYWP4WL3AV6Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681305588 (60h1m9.403868628s) [noise=true , address=77.90.31.15:4720 , piece_id=2FUGVAEJINETPVOX2WODXHIMIPCSPLEVWDGT3W4AGSP4LTE3IZDQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681433672 (60h1m9.403996702s) [noise=true , address=104.36.82.46:4482 , piece_id=75EHPLZR3CHQSHLRSQKSBMWGKCNLZRPIL7R2TJTO3FW34RUY7KOQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681515028 (60h1m9.404078068s) [noise=true , address=45.91.116.5:4307 , piece_id=MDMUYA66DBRD25ERROMFIQSSKKDEG3LS4AYGJQI54LYHHWN64BGQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681655116 (60h1m9.404218156s) [noise=true , address=45.91.116.9:4277 , piece_id=YK7OIYGDM5YST3FNOSVFL3LBZGLDU7UU2LXP3URM74S4BNBWUEJA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681720541 (60h1m9.404283581s) [noise=true , address=77.90.31.12:4602 , piece_id=HMQPZFADUJ73QVP45RDVQ74ZWYLA7Q3ZHEKAITFCGOTDJSZSLZUQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681888082 (60h1m9.404451122s) [noise=true , address=104.36.82.54:4184 , piece_id=BXYUM7HMK4J5IIJJPVK5NXJH3TQE3JLGYAWCPYCITB6E3AM6HKSQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.681951202 (60h1m9.404514242s) [noise=true , address=104.36.82.44:4872 , piece_id=ODYP6BQEMI7G25YWBM6D3JKI4DT5AADQFXONOIAH3SBJTKR7Z44Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682116008 (60h1m9.404679048s) [noise=true , address=64.49.13.138:4483 , piece_id=5P724JSFAKOINUQX733QCCKYUYSTFKVVRMO5HFCLFBJDIPI3JKHQ, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682191392 (60h1m9.404754422s) [noise=true , address=104.36.82.22:4538 , piece_id=UEHEDNUTQ6L2RWTZ25PFYOUIP4LILKWSMKOBBD5FC53TVCMGDHIA, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:36:59.682398979 (60h1m9.404962019s) [noise=true , address=77.90.31.17:4792 , piece_id=2ZA76STMDR63VA24ZZNPJKTE3CPLNRFIGWVAUITFLGRBSRBK7C2Q, seg_pos={0 0}, upload=57585]: Uploading piece... Jul 26 23:37:00.789888640 (60h1m10.51245168s) [upload=57585]: Next returned nil segment Jul 26 23:37:00.789906635 (60h1m10.512469675s) [upload=57585]: Waiting for error group managing segments... Jul 26 23:37:00.956777980 (60h1m10.67934102s) [noise=true , address=64.49.13.138:4483 , piece_id=5P724JSFAKOINUQX733QCCKYUYSTFKVVRMO5HFCLFBJDIPI3JKHQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:37:01.070106992 (60h1m10.792670032s) [noise=true , address=79.127.205.233:4639 , piece_id=MYHLASW5TLXLY6SNYA4GGP4WLMZJ6QEBPGZU3OVHO5GT2WWU4M4A, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.099846674 (60h1m10.822409714s) [noise=true , address=79.127.205.246:4274 , piece_id=YDQ4WQATQITXAGG4BXS6YSDR2BQRIZN3DYH2MFJJVTITQIYM54IQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.135438291 (60h1m10.858001331s) [noise=true , address=79.127.205.235:4677 , piece_id=65OUDRPAZWTNUPPWOAKM6J73GZU7EL3K6UTGEDJ2M24XIGZ22FYA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.591445782 (60h1m11.314008822s) [noise=true , address=45.91.116.9:4277 , piece_id=YK7OIYGDM5YST3FNOSVFL3LBZGLDU7UU2LXP3URM74S4BNBWUEJA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.662548282 (60h1m11.385111322s) [noise=true , address=79.127.219.38:4015 , piece_id=JPSZ3NOQAAYJRKRGPWK3U7HUHD2SU7EFCNACKYE5K7ZE7WOS7DNA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.708791789 (60h1m11.431354829s) [noise=true , address=77.90.31.16:4748 , piece_id=QMC2ZIYYHWQYGJFZGWGU4HE4CGT7DGCCWMJNF7JJKYWP4WL3AV6Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.751670870 (60h1m11.47423391s) [noise=true , address=77.90.31.12:4602 , piece_id=HMQPZFADUJ73QVP45RDVQ74ZWYLA7Q3ZHEKAITFCGOTDJSZSLZUQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:01.772742276 (60h1m11.495305316s) [noise=true , address=77.90.31.18:4821 , piece_id=DJXMXCRUBMOB4SQACXOX6376MUA4CQRWEOJAK3N2E6DO6PFV3IPQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:07.721357463 (60h1m17.443920503s) [noise=true , address=45.91.116.5:4307 , piece_id=MDMUYA66DBRD25ERROMFIQSSKKDEG3LS4AYGJQI54LYHHWN64BGQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: <nil> Jul 26 23:37:58.546719995 (60h2m8.269283035s) [noise=true , address=77.90.31.15:4720 , piece_id=2FUGVAEJINETPVOX2WODXHIMIPCSPLEVWDGT3W4AGSP4LTE3IZDQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:37:58.546880763 (60h2m8.269443803s) [noise=true , address=104.36.82.46:4482 , piece_id=75EHPLZR3CHQSHLRSQKSBMWGKCNLZRPIL7R2TJTO3FW34RUY7KOQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547007456 (60h2m8.269570496s) [noise=true , address=104.36.82.47:4547 , piece_id=NFC2PLVBXUQZTEZZXRP7WMSIOPGJHWFDGHL35Q7NWQ74ZO2RBAVA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547041451 (60h2m8.269604491s) [noise=true , address=77.90.31.17:4792 , piece_id=2ZA76STMDR63VA24ZZNPJKTE3CPLNRFIGWVAUITFLGRBSRBK7C2Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: rpc: dial tcp 77.90.31.17:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547218600 (60h2m8.26978164s) [noise=true , address=104.36.82.22:4538 , piece_id=UEHEDNUTQ6L2RWTZ25PFYOUIP4LILKWSMKOBBD5FC53TVCMGDHIA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:104.36.82.22:4538): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:37:58.547269857 (60h2m8.269832897s) [noise=true , address=104.36.82.21:4493 , piece_id=Q52S6O65QUB3LSC66EPGY62ESVGB7RQZY2W7I6WUZK2Z4ILFDOQA, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: rpc: dial tcp 104.36.82.21:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547316517 (60h2m8.269879557s) [noise=true , address=104.36.82.54:4184 , piece_id=BXYUM7HMK4J5IIJJPVK5NXJH3TQE3JLGYAWCPYCITB6E3AM6HKSQ, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG, address:104.36.82.54:4184): protocol: rpc: dial tcp 104.36.82.54:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547350221 (60h2m8.269913261s) [noise=true , address=104.36.82.44:4872 , piece_id=ODYP6BQEMI7G25YWBM6D3JKI4DT5AADQFXONOIAH3SBJTKR7Z44Q, seg_pos={0 0}, upload=57585]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: write tcp 188.166.225.170:42584->104.36.82.44:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:58.547363888 (60h2m8.269926928s) [seg_pos={0 0}, upload=57585]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:37:58.779170594 (60h2m8.501733634s) [upload=57585]: Done uploading segments... Jul 26 23:37:58.779189010 (60h2m8.50175205s) [upload=57585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.233718097s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)