Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX] duration: 30.238734285s FAIL (2025-07-27 00:26:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.238760544s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X.] duration: 7.083443207s FAIL (2025-07-27 00:25:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.083453888s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [....X......X...XX..XX.X.XX...X...............XX......X...XXX] duration: 1m0.001733751s FAIL (2025-07-27 00:25:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001752537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23187475s FAIL (2025-07-27 00:25:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 97s) [seg_pos={0 0}, upload=58315]: Done begin upload segment. Jul 27 00:24:37.334288710 (60h48m47.056851751s) [noise=true , address=77.90.31.18:4823 , piece_id=75YRBF3BLVSJAY4TS5K4SL23WU66LFNIGMAXT72VY5HP5JUAMHUQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.334307195 (60h48m47.056870235s) [noise=true , address=79.127.205.233:4643 , piece_id=7TXXKKPFCK2NOJVMSUFGHNJANWXSFQDL6DNSSWK2BB4W5YOQXSXA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.334998738 (60h48m47.057561768s) [noise=true , address=104.36.82.46:4509 , piece_id=EGI6YQ7D5DZCRSYOWHMBQJ44B7ZO55I7USYAAJBRA4AZ7AWUPEMQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335530216 (60h48m47.058093256s) [noise=true , address=104.36.82.12:4177 , piece_id=KEO7UTHV5NDQZMZBVUXEHLQNN2PVXG4S6BTAHEZEHVNOVLGFA7LA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335555815 (60h48m47.058118855s) [noise=true , address=79.127.205.234:4608 , piece_id=SJAIIQHHDV6VQMF5X3JZVNHVQFLLOIOUTR42EWQ5VJQQXNEKB2QQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335969636 (60h48m47.058532687s) [noise=true , address=45.91.116.8:4062 , piece_id=ZJRGDD327WI4BOPKX7G5F7GAKIGHUNIITWFF5TGZ4RYJTUYTBWTQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336030713 (60h48m47.058593763s) [noise=true , address=77.90.31.12:4615 , piece_id=RDPLRWF2BXRZAS6CV2LSORT2B6UBJNTQ6MZMA5PT7ZYOS6I4GOAQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336335105 (60h48m47.058898155s) [noise=true , address=79.127.201.215:4346 , piece_id=3I3SATYKAX4K7DIQZ5TMYZXFK5Q345PUI6UZTDNN3HHOUA6KNT2Q, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336425177 (60h48m47.058988218s) [noise=true , address=77.90.31.16:4748 , piece_id=RBZDHET4TF54A76JF53UKJBXFRDK3RSP5GL7EHWEXKXPWP6B52FA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336669966 (60h48m47.059233016s) [noise=true , address=77.90.31.11:4584 , piece_id=5WCJ6OLNF33HHLDERCEADIDQX7LY7HYN4GF3IRPDXHVLTRVYIZZQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336773284 (60h48m47.059336324s) [noise=true , address=79.127.219.48:4074 , piece_id=ZAHQH5XY62MTJZDAYBJ5D3MMTUWMLMWMDA32W3BGBCXPWYVTPKFA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336996962 (60h48m47.059560002s) [noise=true , address=77.90.31.19:4875 , piece_id=NKQMS2ZMA3T5G5RZMXI5LAHKXULK5BLPTFKDRM2NRIYJ2MV7PO3Q, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.337294461 (60h48m47.059857491s) [noise=true , address=79.127.219.38:4004 , piece_id=KOSLKQ4RQMH4JYMKX5CXKFFFFOTSOYGSIAVUJK4NHH62U6EAP2CA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.337598934 (60h48m47.060161974s) [noise=true , address=79.127.205.235:4678 , piece_id=Q2YH2ZQOT3C4YTTUYRWJVTPUPUXWXBXFMZAVLJSU3RXUT6KOXXRA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338009940 (60h48m47.06057298s) [noise=true , address=77.90.31.17:4792 , piece_id=7TFJFPN6MZT44LXORDGKY2T54ML6O2FPJP7E2DLCMJAHPLY2SZYA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338329662 (60h48m47.060892702s) [noise=true , address=104.36.82.47:4522 , piece_id=DPQ5V6I5QWPVB2725PTCTNBML746IO6KUSOIFD5V2BBTCSZO4TAQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338445674 (60h48m47.061008724s) [noise=true , address=79.127.220.97:4361 , piece_id=5WHAGUQN6T73R3MZ5MXZ4Q5VMCPSUAZP7XM3QS6H4AOA6AAYUSNA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338755587 (60h48m47.061318627s) [noise=true , address=77.90.31.15:4722 , piece_id=Q56IQCWRU3NN5RGKZ5T554E5XGZRRGO6HLM3ND2K32SILMCJXZXQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:38.428303890 (60h48m48.15086693s) [upload=58315]: Next returned nil segment Jul 27 00:24:38.428329990 (60h48m48.15089302s) [upload=58315]: Waiting for error group managing segments... Jul 27 00:24:38.714789152 (60h48m48.437352192s) [noise=true , address=79.127.205.233:4643 , piece_id=7TXXKKPFCK2NOJVMSUFGHNJANWXSFQDL6DNSSWK2BB4W5YOQXSXA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.728061286 (60h48m48.450624316s) [noise=true , address=79.127.205.235:4678 , piece_id=Q2YH2ZQOT3C4YTTUYRWJVTPUPUXWXBXFMZAVLJSU3RXUT6KOXXRA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.787440807 (60h48m48.510003847s) [noise=true , address=79.127.205.234:4608 , piece_id=SJAIIQHHDV6VQMF5X3JZVNHVQFLLOIOUTR42EWQ5VJQQXNEKB2QQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.881743626 (60h48m48.604306656s) [noise=true , address=79.127.219.48:4074 , piece_id=ZAHQH5XY62MTJZDAYBJ5D3MMTUWMLMWMDA32W3BGBCXPWYVTPKFA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:127YpFQAmSFbgY2cmAEwcCPDUhg9AnRHS3SkcafEaQeSGVUwVdj, address:79.127.219.48:4074): protocol: not enough available disk space, have: 0, need: 6725632 Jul 27 00:24:39.295262803 (60h48m49.017825833s) [noise=true , address=79.127.219.38:4004 , piece_id=KOSLKQ4RQMH4JYMKX5CXKFFFFOTSOYGSIAVUJK4NHH62U6EAP2CA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:39.321491867 (60h48m49.044054897s) [noise=true , address=77.90.31.18:4823 , piece_id=75YRBF3BLVSJAY4TS5K4SL23WU66LFNIGMAXT72VY5HP5JUAMHUQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:39.398696573 (60h48m49.121259613s) [noise=true , address=77.90.31.15:4722 , piece_id=Q56IQCWRU3NN5RGKZ5T554E5XGZRRGO6HLM3ND2K32SILMCJXZXQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:41.245237621 (60h48m50.967800661s) [noise=true , address=79.127.220.97:4361 , piece_id=5WHAGUQN6T73R3MZ5MXZ4Q5VMCPSUAZP7XM3QS6H4AOA6AAYUSNA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:43.166335509 (60h48m52.888898549s) [noise=true , address=45.91.116.8:4062 , piece_id=ZJRGDD327WI4BOPKX7G5F7GAKIGHUNIITWFF5TGZ4RYJTUYTBWTQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, address:45.91.116.8:4062): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:47.603476894 (60h48m57.326039934s) [noise=true , address=77.90.31.19:4875 , piece_id=NKQMS2ZMA3T5G5RZMXI5LAHKXULK5BLPTFKDRM2NRIYJ2MV7PO3Q, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:02.743320214 (60h49m12.465883255s) [noise=true , address=79.127.201.215:4346 , piece_id=3I3SATYKAX4K7DIQZ5TMYZXFK5Q345PUI6UZTDNN3HHOUA6KNT2Q, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:23.706987212 (60h49m33.429550252s) [noise=true , address=77.90.31.12:4615 , piece_id=RDPLRWF2BXRZAS6CV2LSORT2B6UBJNTQ6MZMA5PT7ZYOS6I4GOAQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:36.225934902 (60h49m45.948497942s) [noise=true , address=77.90.31.16:4748 , piece_id=RBZDHET4TF54A76JF53UKJBXFRDK3RSP5GL7EHWEXKXPWP6B52FA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226054981 (60h49m45.948618021s) [noise=true , address=77.90.31.11:4584 , piece_id=5WCJ6OLNF33HHLDERCEADIDQX7LY7HYN4GF3IRPDXHVLTRVYIZZQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:36.226500133 (60h49m45.949063173s) [noise=true , address=104.36.82.12:4177 , piece_id=KEO7UTHV5NDQZMZBVUXEHLQNN2PVXG4S6BTAHEZEHVNOVLGFA7LA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:104.36.82.12:4177): protocol: write tcp 188.166.225.170:44790->104.36.82.12:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226655220 (60h49m45.94921827s) [noise=true , address=77.90.31.17:4792 , piece_id=7TFJFPN6MZT44LXORDGKY2T54ML6O2FPJP7E2DLCMJAHPLY2SZYA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: read tcp 188.166.225.170:40740->77.90.31.17:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226681340 (60h49m45.94924439s) [noise=true , address=104.36.82.47:4522 , piece_id=DPQ5V6I5QWPVB2725PTCTNBML746IO6KUSOIFD5V2BBTCSZO4TAQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226701448 (60h49m45.949264488s) [noise=true , address=104.36.82.46:4509 , piece_id=EGI6YQ7D5DZCRSYOWHMBQJ44B7ZO55I7USYAAJBRA4AZ7AWUPEMQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:38690->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226733660 (60h49m45.9492967s) [seg_pos={0 0}, upload=58315]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:25:36.455809672 (60h49m46.178372702s) [upload=58315]: Done uploading segments... Jul 27 00:25:36.455833917 (60h49m46.178396957s) [upload=58315]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231886733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXX] duration: 30.234363344s FAIL (2025-07-27 00:25:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23440325s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.] duration: 5.945013099s FAIL (2025-07-27 00:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 699 (60h48m30.885503729s) [upload=58310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:24:21.455742386 (60h48m31.178305426s) [upload=58310]: Done uploading segments... Jul 27 00:24:21.455761042 (60h48m31.178324082s) [upload=58310]: Upload finished. err: <nil> Jul 27 00:24:49.608066787 (60h48m59.330629827s) [upload=58318]: Starting upload Jul 27 00:24:49.608140568 (60h48m59.330703608s) [upload=58318]: Done starting upload Jul 27 00:24:49.608229108 (60h48m59.330792148s) [upload=58318]: Uploading segments... Jul 27 00:24:49.608362062 (60h48m59.330925102s) [upload=58318]: Got next segment. Inline: false Jul 27 00:24:49.608391879 (60h48m59.330954919s) [upload=58318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:49.608396478 (60h48m59.330959508s) [upload=58318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:49.844478273 (60h48m59.567041313s) [seg_pos={0 0}, upload=58318]: Begin upload segment... Jul 27 00:24:49.844522468 (60h48m59.567085498s) [seg_pos={0 0}, upload=58318]: Done begin upload segment. Jul 27 00:24:49.844555762 (60h48m59.567118792s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.844978750 (60h48m59.56754178s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845300717 (60h48m59.567863757s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845584860 (60h48m59.5681479s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845871880 (60h48m59.56843492s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846186351 (60h48m59.568749391s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846458602 (60h48m59.569021642s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847020477 (60h48m59.569583527s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847638309 (60h48m59.570201339s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847850616 (60h48m59.570413656s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848056950 (60h48m59.57061998s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848258606 (60h48m59.570821646s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848441055 (60h48m59.571004095s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848642931 (60h48m59.571205971s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848821513 (60h48m59.571384553s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.849008110 (60h48m59.57157115s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:51.509939290 (60h49m1.23250233s) [upload=58318]: Next returned nil segment Jul 27 00:24:51.509951304 (60h49m1.232514334s) [upload=58318]: Waiting for error group managing segments... Jul 27 00:24:51.848825802 (60h49m1.571388842s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.868516997 (60h49m1.591080037s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.870867164 (60h49m1.593430194s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.940969901 (60h49m1.663532981s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.948408754 (60h49m1.670971834s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.949260253 (60h49m1.671823293s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.958460255 (60h49m1.681023305s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.959566633 (60h49m1.682129673s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:53.420609046 (60h49m3.143172086s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:25:18.961139860 (60h49m28.6837029s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961158235 (60h49m28.683721275s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961196839 (60h49m28.683759879s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961212729 (60h49m28.683775769s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961239480 (60h49m28.68380252s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961252716 (60h49m28.683815746s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:58116->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961277724 (60h49m28.683840754s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961303412 (60h49m28.683866452s) [seg_pos={0 0}, upload=58318]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:25:19.188870085 (60h49m28.911433115s) [upload=58318]: Done uploading segments... Jul 27 00:25:19.188893971 (60h49m28.911457001s) [upload=58318]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration:* 5.945031744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X] duration: 30.229867622s FAIL (2025-07-27 00:25:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 699 (60h48m30.885503729s) [upload=58310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:24:21.455742386 (60h48m31.178305426s) [upload=58310]: Done uploading segments... Jul 27 00:24:21.455761042 (60h48m31.178324082s) [upload=58310]: Upload finished. err: <nil> Jul 27 00:24:49.608066787 (60h48m59.330629827s) [upload=58318]: Starting upload Jul 27 00:24:49.608140568 (60h48m59.330703608s) [upload=58318]: Done starting upload Jul 27 00:24:49.608229108 (60h48m59.330792148s) [upload=58318]: Uploading segments... Jul 27 00:24:49.608362062 (60h48m59.330925102s) [upload=58318]: Got next segment. Inline: false Jul 27 00:24:49.608391879 (60h48m59.330954919s) [upload=58318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:49.608396478 (60h48m59.330959508s) [upload=58318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:49.844478273 (60h48m59.567041313s) [seg_pos={0 0}, upload=58318]: Begin upload segment... Jul 27 00:24:49.844522468 (60h48m59.567085498s) [seg_pos={0 0}, upload=58318]: Done begin upload segment. Jul 27 00:24:49.844555762 (60h48m59.567118792s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.844978750 (60h48m59.56754178s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845300717 (60h48m59.567863757s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845584860 (60h48m59.5681479s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845871880 (60h48m59.56843492s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846186351 (60h48m59.568749391s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846458602 (60h48m59.569021642s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847020477 (60h48m59.569583527s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847638309 (60h48m59.570201339s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847850616 (60h48m59.570413656s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848056950 (60h48m59.57061998s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848258606 (60h48m59.570821646s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848441055 (60h48m59.571004095s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848642931 (60h48m59.571205971s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848821513 (60h48m59.571384553s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.849008110 (60h48m59.57157115s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:51.509939290 (60h49m1.23250233s) [upload=58318]: Next returned nil segment Jul 27 00:24:51.509951304 (60h49m1.232514334s) [upload=58318]: Waiting for error group managing segments... Jul 27 00:24:51.848825802 (60h49m1.571388842s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.868516997 (60h49m1.591080037s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.870867164 (60h49m1.593430194s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.940969901 (60h49m1.663532981s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.948408754 (60h49m1.670971834s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.949260253 (60h49m1.671823293s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.958460255 (60h49m1.681023305s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.959566633 (60h49m1.682129673s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:53.420609046 (60h49m3.143172086s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:25:18.961139860 (60h49m28.6837029s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961158235 (60h49m28.683721275s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961196839 (60h49m28.683759879s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961212729 (60h49m28.683775769s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961239480 (60h49m28.68380252s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961252716 (60h49m28.683815746s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:58116->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961277724 (60h49m28.683840754s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961303412 (60h49m28.683866452s) [seg_pos={0 0}, upload=58318]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:25:19.188870085 (60h49m28.911433115s) [upload=58318]: Done uploading segments... Jul 27 00:25:19.188893971 (60h49m28.911457001s) [upload=58318]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration:* 30.229886599s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X] duration: 30.001082659s FAIL (2025-07-27 00:25:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00110952s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................................X.] duration: 1.866363284s FAIL (2025-07-27 00:25:05 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.86639745s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXX] duration: 30.234269907s FAIL (2025-07-27 00:24:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.234289205s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X......X...XX..XX.X.XX...X...............XX......X...XX] duration: 1m0.001063366s FAIL (2025-07-27 00:24:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001075339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.25533714s FAIL (2025-07-27 00:24:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 4ITBHVQTFB2TPBOQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.837789598 (60h47m46.560352638s) [noise=true , address=79.127.219.47:4048 , piece_id=3C2ITTUR72D6BNWJHSACLD5FVSAWWMRCANC2YHYOHWSX6IFNWX5A, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.837922892 (60h47m46.560485932s) [noise=true , address=104.36.82.46:4508 , piece_id=ZKWZ66J253ERR6CBZZ3NBLX5TJUSUINWDKD7E2NBTPIUT2B5XWYQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838073811 (60h47m46.560636841s) [noise=true , address=79.127.205.246:4284 , piece_id=EWBYPOYB57TF7ESYKFZDW3U6UGO7DAJFZHM7BBPY26F44N2BHYQA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838326424 (60h47m46.560889464s) [noise=true , address=79.127.201.215:4358 , piece_id=QDVE6IR4ZANX7SXQPF7QUXBXGAPS2MCROSNHZEHTED55AGQD3BMQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838561735 (60h47m46.561124775s) [noise=true , address=104.36.82.11:4130 , piece_id=EOUTV7ZFMCRBCR5ZNS7HMPNCKPKSBP6GTUOIKDPORMOBGA2BPHJA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839328331 (60h47m46.561891371s) [noise=true , address=104.36.82.53:4131 , piece_id=M5BCO2IMSUTIFK6UWCNVXR5GPN6OQGI3JJDRMZNHFONM24AZBETA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839588259 (60h47m46.562151299s) [noise=true , address=79.127.205.234:4619 , piece_id=6E56L4RSFUFBVNHAG7VOAPLDX7TPTSRAKZR2ZNTOIZ3H4XAS3BCA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839865719 (60h47m46.562428759s) [noise=true , address=77.90.31.13:4659 , piece_id=CPK7G6RSRP5REL4XEKQFPDZK4NWA4SFL2H4LWKQGGGUM7FTQSMHA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840168970 (60h47m46.56273201s) [noise=true , address=104.36.82.56:4230 , piece_id=W3XFY346SXAI5HGYCPPC74TJOEIDCTZXD2X55XDRYYTYB5WWNRGQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840431182 (60h47m46.562994222s) [noise=true , address=45.91.116.14:4206 , piece_id=QNJ4IC7JYAJAOEH52YOTPCNU56O6UYJYUVKCP24NMSVGAYQHVDLQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840459636 (60h47m46.563022676s) [noise=true , address=79.127.205.245:4242 , piece_id=MESWK4LX4HZLIYBNFMVWRLJ23EIYUQFZP3GHYVCFL5KZ4TA3MFBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840823823 (60h47m46.563386863s) [noise=true , address=104.36.82.52:4091 , piece_id=CDSJXFEQMOHFET66QNSSUZ3TTJGPJ4DL2QEBZIO5TMBCMQSFUQPA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841195363 (60h47m46.563758403s) [noise=true , address=104.36.82.7:4918 , piece_id=4UP4UL5KEJB3WAR255LPD7QCLDDJFTJZUG4AYQHRHEPHTROQ3XMA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841510827 (60h47m46.564073867s) [noise=true , address=64.49.13.138:4488 , piece_id=E4ZHOT6QY4L57Z7UBGEEDQSRDYUSMLVTFHOEVE4BMPOC5JEJR3NA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841762979 (60h47m46.564326019s) [noise=true , address=79.127.205.235:4684 , piece_id=GNEQTGPWROAIRJARUD5T5PHG2M4BEMSNJFOEE3RYVMS3COYRD74Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.842054518 (60h47m46.564617568s) [noise=true , address=104.36.82.47:4532 , piece_id=AL35UOUZQF46QRL6LS5NRJPS6YETX3JWD3WL2GFM4CFLTHLY6NNA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:37.940047468 (60h47m47.662610508s) [upload=58299]: Next returned nil segment Jul 27 00:23:37.940062065 (60h47m47.662625105s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:23:38.033467017 (60h47m47.756030057s) [noise=true , address=79.127.219.47:4048 , piece_id=3C2ITTUR72D6BNWJHSACLD5FVSAWWMRCANC2YHYOHWSX6IFNWX5A, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:23:38.249706559 (60h47m47.972269599s) [noise=true , address=79.127.205.245:4242 , piece_id=MESWK4LX4HZLIYBNFMVWRLJ23EIYUQFZP3GHYVCFL5KZ4TA3MFBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.262287289 (60h47m47.984850329s) [noise=true , address=79.127.205.246:4284 , piece_id=EWBYPOYB57TF7ESYKFZDW3U6UGO7DAJFZHM7BBPY26F44N2BHYQA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.263857663 (60h47m47.986420704s) [noise=true , address=79.127.205.235:4684 , piece_id=GNEQTGPWROAIRJARUD5T5PHG2M4BEMSNJFOEE3RYVMS3COYRD74Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.300331980 (60h47m48.02289502s) [noise=true , address=79.127.205.234:4619 , piece_id=6E56L4RSFUFBVNHAG7VOAPLDX7TPTSRAKZR2ZNTOIZ3H4XAS3BCA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:39.264427374 (60h47m48.986990414s) [noise=true , address=45.91.116.14:4206 , piece_id=QNJ4IC7JYAJAOEH52YOTPCNU56O6UYJYUVKCP24NMSVGAYQHVDLQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:39.296307862 (60h47m49.018870902s) [noise=true , address=64.49.13.138:4488 , piece_id=E4ZHOT6QY4L57Z7UBGEEDQSRDYUSMLVTFHOEVE4BMPOC5JEJR3NA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:51.816611526 (60h48m1.539174666s) [noise=true , address=79.127.201.215:4358 , piece_id=QDVE6IR4ZANX7SXQPF7QUXBXGAPS2MCROSNHZEHTED55AGQD3BMQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:22.406894987 (60h48m32.129458027s) [noise=true , address=77.90.31.13:4659 , piece_id=CPK7G6RSRP5REL4XEKQFPDZK4NWA4SFL2H4LWKQGGGUM7FTQSMHA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:35.737981876 (60h48m45.460544906s) [noise=true , address=104.36.82.53:4131 , piece_id=M5BCO2IMSUTIFK6UWCNVXR5GPN6OQGI3JJDRMZNHFONM24AZBETA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: rpc: dial tcp 104.36.82.53:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738027644 (60h48m45.460590684s) [noise=true , address=104.36.82.11:4130 , piece_id=EOUTV7ZFMCRBCR5ZNS7HMPNCKPKSBP6GTUOIKDPORMOBGA2BPHJA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12n69riZBquh3Qi436LVfVgNDJYuDxPzGi7d2BZDLurUJYVMCqs, address:104.36.82.11:4130): protocol: rpc: dial tcp 104.36.82.11:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738124469 (60h48m45.460687499s) [noise=true , address=104.36.82.52:4091 , piece_id=CDSJXFEQMOHFET66QNSSUZ3TTJGPJ4DL2QEBZIO5TMBCMQSFUQPA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738159026 (60h48m45.460722066s) [noise=true , address=104.36.82.7:4918 , piece_id=4UP4UL5KEJB3WAR255LPD7QCLDDJFTJZUG4AYQHRHEPHTROQ3XMA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 188.166.225.170:37276->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738190816 (60h48m45.460753856s) [noise=true , address=77.90.31.17:4774 , piece_id=55FG6E6IV64EKCLFYVYUCFKU24RRSVQXTAAC4ITBHVQTFB2TPBOQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: read tcp 188.166.225.170:45818->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738195054 (60h48m45.460758094s) [noise=true , address=104.36.82.46:4508 , piece_id=ZKWZ66J253ERR6CBZZ3NBLX5TJUSUINWDKD7E2NBTPIUT2B5XWYQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:24:35.738239961 (60h48m45.460803001s) [noise=true , address=104.36.82.47:4532 , piece_id=AL35UOUZQF46QRL6LS5NRJPS6YETX3JWD3WL2GFM4CFLTHLY6NNA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:24:35.738252795 (60h48m45.460815835s) [noise=true , address=104.36.82.56:4230 , piece_id=W3XFY346SXAI5HGYCPPC74TJOEIDCTZXD2X55XDRYYTYB5WWNRGQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738283493 (60h48m45.460846533s) [noise=true , address=77.90.31.16:4747 , piece_id=UGSF7DBZUBBKNK7V5KAO5JR7ZMJAQMLHD6BR6WPLUG4UMZHWYHZQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: rpc: dial tcp 77.90.31.16:4747: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738298953 (60h48m45.460861993s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:24:35.991568886 (60h48m45.714131926s) [upload=58299]: Done uploading segments... Jul 27 00:24:35.991593102 (60h48m45.714156142s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.255352441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXX] duration: 30.242511869s FAIL (2025-07-27 00:24:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24253325s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.] duration: 7.116922434s FAIL (2025-07-27 00:24:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== FAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:23:26.878833834 (60h47m36.601396864s) [upload=58298]: Starting upload Jul 27 00:23:26.878889149 (60h47m36.60145219s) [upload=58298]: Done starting upload Jul 27 00:23:26.878958201 (60h47m36.601521241s) [upload=58298]: Uploading segments... Jul 27 00:23:26.879205003 (60h47m36.601768043s) [upload=58298]: Got next segment. Inline: false Jul 27 00:23:26.879232696 (60h47m36.601795736s) [upload=58298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:26.879237807 (60h47m36.601800887s) [upload=58298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:27.117701870 (60h47m36.84026491s) [seg_pos={0 0}, upload=58298]: Begin upload segment... Jul 27 00:23:27.117772716 (60h47m36.840335756s) [seg_pos={0 0}, upload=58298]: Done begin upload segment. Jul 27 00:23:27.117840324 (60h47m36.840403364s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.117860343 (60h47m36.840423383s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118423991 (60h47m36.840987031s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118698997 (60h47m36.841262037s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119055539 (60h47m36.841618579s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119302452 (60h47m36.841865482s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119503998 (60h47m36.842067028s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119695213 (60h47m36.842258253s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119894365 (60h47m36.842457405s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120153350 (60h47m36.84271639s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120380224 (60h47m36.842943254s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120589725 (60h47m36.843152765s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120830175 (60h47m36.843393215s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121036910 (60h47m36.84359995s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121246862 (60h47m36.843809892s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121523792 (60h47m36.844086822s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:28.854018534 (60h47m38.576581564s) [upload=58298]: Next returned nil segment Jul 27 00:23:28.854052358 (60h47m38.576615398s) [upload=58298]: Waiting for error group managing segments... Jul 27 00:23:29.129523117 (60h47m38.852086157s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155007436 (60h47m38.877570476s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155488357 (60h47m38.878051397s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.157090652 (60h47m38.879653692s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.222674814 (60h47m38.945237854s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.223964532 (60h47m38.946527572s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.232334866 (60h47m38.954897907s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.236272311 (60h47m38.958835361s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.302653628 (60h47m39.025216708s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.356275302 (60h47m39.078838332s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:31.238097536 (60h47m40.960660576s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:56.226454714 (60h48m5.949017754s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:51898->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226501363 (60h48m5.949064403s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226622195 (60h48m5.949185235s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39816->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226649257 (60h48m5.949212297s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226667200 (60h48m5.94923024s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226683812 (60h48m5.949246852s) [seg_pos={0 0}, upload=58298]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:23:56.458682075 (60h48m6.181245115s) [upload=58298]: Done uploading segments... Jul 27 00:23:56.458702984 (60h48m6.181266025s) [upload=58298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.116946749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 1.39121363s FAIL (2025-07-27 00:23:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.391241093s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX] duration: 30.233287166s FAIL (2025-07-27 00:23:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== FAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:23:26.878833834 (60h47m36.601396864s) [upload=58298]: Starting upload Jul 27 00:23:26.878889149 (60h47m36.60145219s) [upload=58298]: Done starting upload Jul 27 00:23:26.878958201 (60h47m36.601521241s) [upload=58298]: Uploading segments... Jul 27 00:23:26.879205003 (60h47m36.601768043s) [upload=58298]: Got next segment. Inline: false Jul 27 00:23:26.879232696 (60h47m36.601795736s) [upload=58298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:26.879237807 (60h47m36.601800887s) [upload=58298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:27.117701870 (60h47m36.84026491s) [seg_pos={0 0}, upload=58298]: Begin upload segment... Jul 27 00:23:27.117772716 (60h47m36.840335756s) [seg_pos={0 0}, upload=58298]: Done begin upload segment. Jul 27 00:23:27.117840324 (60h47m36.840403364s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.117860343 (60h47m36.840423383s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118423991 (60h47m36.840987031s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118698997 (60h47m36.841262037s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119055539 (60h47m36.841618579s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119302452 (60h47m36.841865482s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119503998 (60h47m36.842067028s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119695213 (60h47m36.842258253s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119894365 (60h47m36.842457405s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120153350 (60h47m36.84271639s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120380224 (60h47m36.842943254s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120589725 (60h47m36.843152765s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120830175 (60h47m36.843393215s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121036910 (60h47m36.84359995s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121246862 (60h47m36.843809892s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121523792 (60h47m36.844086822s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:28.854018534 (60h47m38.576581564s) [upload=58298]: Next returned nil segment Jul 27 00:23:28.854052358 (60h47m38.576615398s) [upload=58298]: Waiting for error group managing segments... Jul 27 00:23:29.129523117 (60h47m38.852086157s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155007436 (60h47m38.877570476s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155488357 (60h47m38.878051397s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.157090652 (60h47m38.879653692s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.222674814 (60h47m38.945237854s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.223964532 (60h47m38.946527572s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.232334866 (60h47m38.954897907s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.236272311 (60h47m38.958835361s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.302653628 (60h47m39.025216708s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.356275302 (60h47m39.078838332s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:31.238097536 (60h47m40.960660576s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:56.226454714 (60h48m5.949017754s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:51898->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226501363 (60h48m5.949064403s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226622195 (60h48m5.949185235s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39816->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226649257 (60h48m5.949212297s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226667200 (60h48m5.94923024s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226683812 (60h48m5.949246852s) [seg_pos={0 0}, upload=58298]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:23:56.458682075 (60h48m6.181245115s) [upload=58298]: Done uploading segments... Jul 27 00:23:56.458702984 (60h48m6.181266025s) [upload=58298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.233300991s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........] duration: 8.186698978s FAIL (2025-07-27 00:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.186719747s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.........] duration: 28.790575933s FAIL (2025-07-27 00:23:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.790597585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245761056s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== QOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.358889046 (60h46m46.081452086s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359306165 (60h46m46.081869215s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359532648 (60h46m46.082095688s) [seg_pos={0 0}, upload=58286]: Done begin upload segment. Jul 27 00:22:36.359558287 (60h46m46.082121327s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.360857503 (60h46m46.083420543s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361150514 (60h46m46.083713554s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361476588 (60h46m46.084039628s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361767975 (60h46m46.084331015s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361974490 (60h46m46.08453753s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362101503 (60h46m46.084664543s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362583535 (60h46m46.085146575s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362912635 (60h46m46.085475675s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362921011 (60h46m46.085484051s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362956859 (60h46m46.085519899s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363233869 (60h46m46.085796909s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363246133 (60h46m46.085809173s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363537079 (60h46m46.086100119s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363868323 (60h46m46.086431363s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:37.447665282 (60h46m47.170228332s) [upload=58286]: Next returned nil segment Jul 27 00:22:37.447692194 (60h46m47.170255234s) [upload=58286]: Waiting for error group managing segments... Jul 27 00:22:37.501411946 (60h46m47.223974986s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:22:37.745347919 (60h46m47.467910959s) [noise=true , address=79.127.205.234:4606 , piece_id=SYDVXXEWEHM25RAQEGWH47HFZZ7ZOHXDMNQOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.747484648 (60h46m47.470047688s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.781937336 (60h46m47.504500376s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.030972264 (60h46m47.753535304s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.250917325 (60h46m47.973480365s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.332626600 (60h46m48.05518964s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:40.304136871 (60h46m50.026699911s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:41.704307309 (60h46m51.426870349s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:49.259512908 (60h46m58.982075948s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:52.112812417 (60h47m1.835375457s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:23:35.254588588 (60h47m44.977151628s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254765656 (60h47m44.977328686s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254823567 (60h47m44.977386607s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254862192 (60h47m44.977425222s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 188.166.225.170:53120->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254922136 (60h47m44.977485166s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254977973 (60h47m44.977541013s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: rpc: dial tcp 77.90.31.10:4535: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255018531 (60h47m44.977581571s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: write tcp 188.166.225.170:51642->77.90.31.11:4584: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255039040 (60h47m44.97760207s) [seg_pos={0 0}, upload=58286]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:23:35.498418277 (60h47m45.220981307s) [upload=58286]: Done uploading segments... Jul 27 00:23:35.498445649 (60h47m45.221008689s) [upload=58286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.245775625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.....X......X...XX..XX.X.XX...X...............XX......X...X] duration: 1m0.001072982s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001100766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXX] duration: 30.233196508s FAIL (2025-07-27 00:23:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.233223029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXX] duration: 30.234025965s FAIL (2025-07-27 00:23:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== n closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301966701 (60h47m5.024529751s) [seg_pos={0 0}, upload=58285]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:55.535080726 (60h47m5.257643776s) [upload=58285]: Done uploading segments... Jul 27 00:22:55.535111805 (60h47m5.257674845s) [upload=58285]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:22:56.419177168 (60h47m6.141740228s) [upload=58291]: Starting upload Jul 27 00:22:56.419245739 (60h47m6.141808779s) [upload=58291]: Done starting upload Jul 27 00:22:56.419303629 (60h47m6.14186667s) [upload=58291]: Uploading segments... Jul 27 00:22:56.419440371 (60h47m6.142003411s) [upload=58291]: Got next segment. Inline: false Jul 27 00:22:56.419485077 (60h47m6.142048117s) [upload=58291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:56.419489145 (60h47m6.142052185s) [upload=58291]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:56.658833663 (60h47m6.381396703s) [seg_pos={0 0}, upload=58291]: Begin upload segment... Jul 27 00:22:56.658891664 (60h47m6.381454704s) [seg_pos={0 0}, upload=58291]: Done begin upload segment. Jul 27 00:22:56.658915349 (60h47m6.381478389s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658926670 (60h47m6.381489721s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658967740 (60h47m6.38153078s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659311056 (60h47m6.381874086s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659389747 (60h47m6.381952787s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659789111 (60h47m6.382352151s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660189837 (60h47m6.382752877s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660536962 (60h47m6.383100012s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660932729 (60h47m6.383495769s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661252481 (60h47m6.383815521s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661491148 (60h47m6.384054188s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661820648 (60h47m6.384383688s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662040749 (60h47m6.384603789s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662292952 (60h47m6.384855992s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662324492 (60h47m6.384887542s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662640596 (60h47m6.385203636s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:58.385865324 (60h47m8.108428364s) [upload=58291]: Next returned nil segment Jul 27 00:22:58.385883758 (60h47m8.108446798s) [upload=58291]: Waiting for error group managing segments... Jul 27 00:22:58.662217739 (60h47m8.384780779s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.682691902 (60h47m8.405254942s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.683459410 (60h47m8.40602244s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.692085685 (60h47m8.414648725s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.758650774 (60h47m8.481213814s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.762657199 (60h47m8.485220239s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.829026684 (60h47m8.551589724s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.855458056 (60h47m8.578021086s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.878029681 (60h47m8.600592721s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:00.182861607 (60h47m9.905424647s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:25.766371949 (60h47m35.488934989s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:36976->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766427906 (60h47m35.488990936s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766545671 (60h47m35.489108711s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766580197 (60h47m35.489143237s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:52150->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766639831 (60h47m35.489202871s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 188.166.225.170:47750->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766687352 (60h47m35.489250432s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.234042968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.833626699s FAIL (2025-07-27 00:23:13 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833656737s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXX] duration: 30.235907451s FAIL (2025-07-27 00:22:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 50860->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994355370 (60h46m15.71691841s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:22:05.994375438 (60h46m15.716938479s) [seg_pos={0 0}, upload=58272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:22:06.225001157 (60h46m15.947564197s) [upload=58272]: Done uploading segments... Jul 27 00:22:06.225029662 (60h46m15.947592702s) [upload=58272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:22:25.944858424 (60h46m35.667421454s) [upload=58285]: Starting upload Jul 27 00:22:25.944914481 (60h46m35.667477521s) [upload=58285]: Done starting upload Jul 27 00:22:25.945028309 (60h46m35.667591349s) [upload=58285]: Uploading segments... Jul 27 00:22:25.945194817 (60h46m35.667757857s) [upload=58285]: Got next segment. Inline: false Jul 27 00:22:25.945239253 (60h46m35.667802283s) [upload=58285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:25.945244342 (60h46m35.667807382s) [upload=58285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:26.178196499 (60h46m35.900759539s) [seg_pos={0 0}, upload=58285]: Begin upload segment... Jul 27 00:22:26.178252647 (60h46m35.900815687s) [seg_pos={0 0}, upload=58285]: Done begin upload segment. Jul 27 00:22:26.178273276 (60h46m35.900836317s) [noise=true , address=104.36.82.14:4221 , piece_id=TOSIDO6LOV6ZKD4GJHHQ5ZJVDT6NO3QYF7R5QUDIP3ROL7L5BYGA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.178620932 (60h46m35.901183962s) [noise=true , address=45.91.116.8:4045 , piece_id=XCCFCCDKOQAEEPZOF2DYJJFX6LEIGA5W6GSOHPNROPFLWFZNURWQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.178878284 (60h46m35.901441324s) [noise=true , address=77.90.31.15:4699 , piece_id=JZAQWBZ2KYCJRPM3ATBARY5BYFA73EZYTTE4CA5D75VC7MA5IZNQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179154773 (60h46m35.901717813s) [noise=true , address=45.91.116.12:4225 , piece_id=MYW4BZYGMCI6UJUKD7GWIOK6DXVEKBYVY5KBE5XEJX2DH6YQCM2Q, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179431232 (60h46m35.901994262s) [noise=true , address=77.90.31.13:4626 , piece_id=RI7XNQ4QTZMKY5R5OOIC5EZ2Q4S5UJGSEMJGRMRYFSAM5CSXXIRQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179672303 (60h46m35.902235333s) [noise=true , address=45.91.116.13:4153 , piece_id=TYQ3XN3MKX3MCCWKZ7EDZL63EXU4FLZF76C2AUBFQPNBPPXDAOKQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179916270 (60h46m35.90247931s) [noise=true , address=77.90.31.17:4771 , piece_id=XBGFPEY2UH76YR7X24BAAYOTL7FKQCIZZPRP5QXHI3ZLLZOZDUAQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180229179 (60h46m35.902792219s) [noise=true , address=104.36.82.5:4839 , piece_id=DSKFFCUDEKT6OLFW5C7OIAUCOZLDNADRMUAQG2ESI4NMFFZJYCSQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180454760 (60h46m35.90301779s) [noise=true , address=104.36.82.9:4006 , piece_id=SQXU2TH66A57KSAM3DOZUXLK642ALKFSPCGD4U223RB3SRTDNDCA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180660795 (60h46m35.903223835s) [noise=true , address=45.91.116.5:4297 , piece_id=LMPWPXW7ITIIHRRSHULGNACRQ5PKLMPIYVBP4GOEI5YDKEVPKQIQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180890554 (60h46m35.903453594s) [noise=true , address=77.90.31.12:4590 , piece_id=J4B75TSVQ4Q5J2ULRT7XGJDFFXJMSCSQLGLTMQFKWJIWMCRIWWBQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181107900 (60h46m35.90367094s) [noise=true , address=45.91.116.6:4974 , piece_id=YRFYE64PQ43RUINDBVERBOPNUADHPWCVWZZAG7LO6M36KVHMJS4Q, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181360132 (60h46m35.903923172s) [noise=true , address=45.91.116.4:4937 , piece_id=2AXYYLFCUIZA4AZCK2N4OGF36XH4RJWOU6LI2M244MBJDS5YOWNA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181569273 (60h46m35.904132313s) [noise=true , address=77.90.31.19:4842 , piece_id=D2FQTUUCLPBBPCXANQEJE4NORTUXK7MSRTEFKHQZNVAILCBXHEFQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.182070782 (60h46m35.904633812s) [noise=true , address=45.91.116.10:4081 , piece_id=A2TMO3SWLLVJWIIUPR7V5AW63JBAQYV4KEZNPTSF6CBVH64DI54A, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.182540371 (60h46m35.905103411s) [noise=true , address=45.91.116.7:4009 , piece_id=644K4VRNKEAR2PP5JBKNQSNFLVDBYH7KWPNGOPZRMNVIRDLMTEVA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:27.910590149 (60h46m37.633153189s) [upload=58285]: Next returned nil segment Jul 27 00:22:27.910609567 (60h46m37.633172607s) [upload=58285]: Waiting for error group managing segments... Jul 27 00:22:28.199559705 (60h46m37.922122745s) [noise=true , address=45.91.116.6:4974 , piece_id=YRFYE64PQ43RUINDBVERBOPNUADHPWCVWZZAG7LO6M36KVHMJS4Q, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.208008390 (60h46m37.93057142s) [noise=true , address=45.91.116.4:4937 , piece_id=2AXYYLFCUIZA4AZCK2N4OGF36XH4RJWOU6LI2M244MBJDS5YOWNA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.215670731 (60h46m37.938233771s) [noise=true , address=45.91.116.13:4153 , piece_id=TYQ3XN3MKX3MCCWKZ7EDZL63EXU4FLZF76C2AUBFQPNBPPXDAOKQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.259150630 (60h46m37.98171367s) [noise=true , address=45.91.116.5:4297 , piece_id=LMPWPXW7ITIIHRRSHULGNACRQ5PKLMPIYVBP4GOEI5YDKEVPKQIQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.283540055 (60h46m38.006103095s) [noise=true , address=45.91.116.8:4045 , piece_id=XCCFCCDKOQAEEPZOF2DYJJFX6LEIGA5W6GSOHPNROPFLWFZNURWQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.301075536 (60h46m38.023638576s) [noise=true , address=77.90.31.15:4699 , piece_id=JZAQWBZ2KYCJRPM3ATBARY5BYFA73EZYTTE4CA5D75VC7MA5IZNQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.303189240 (60h46m38.02575228s) [noise=true , address=45.91.116.7:4009 , piece_id=644K4VRNKEAR2PP5JBKNQSNFLVDBYH7KWPNGOPZRMNVIRDLMTEVA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.342583963 (60h46m38.065147003s) [noise=true , address=45.91.116.12:4225 , piece_id=MYW4BZYGMCI6UJUKD7GWIOK6DXVEKBYVY5KBE5XEJX2DH6YQCM2Q, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.371911185 (60h46m38.094474225s) [noise=true , address=45.91.116.10:4081 , piece_id=A2TMO3SWLLVJWIIUPR7V5AW63JBAQYV4KEZNPTSF6CBVH64DI54A, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.697862379 (60h46m39.42042542s) [noise=true , address=77.90.31.13:4626 , piece_id=RI7XNQ4QTZMKY5R5OOIC5EZ2Q4S5UJGSEMJGRMRYFSAM5CSXXIRQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.749540456 (60h46m39.472103496s) [noise=true , address=77.90.31.19:4842 , piece_id=D2FQTUUCLPBBPCXANQEJE4NORTUXK7MSRTEFKHQZNVAILCBXHEFQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.891724216 (60h46m39.614287256s) [noise=true , address=77.90.31.12:4590 , piece_id=J4B75TSVQ4Q5J2ULRT7XGJDFFXJMSCSQLGLTMQFKWJIWMCRIWWBQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:55.301704308 (60h47m5.024267348s) [noise=true , address=104.36.82.5:4839 , piece_id=DSKFFCUDEKT6OLFW5C7OIAUCOZLDNADRMUAQG2ESI4NMFFZJYCSQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:44334->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301755897 (60h47m5.024318937s) [noise=true , address=77.90.31.17:4771 , piece_id=XBGFPEY2UH76YR7X24BAAYOTL7FKQCIZZPRP5QXHI3ZLLZOZDUAQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301790393 (60h47m5.024353433s) [noise=true , address=104.36.82.9:4006 , piece_id=SQXU2TH66A57KSAM3DOZUXLK642ALKFSPCGD4U223RB3SRTDNDCA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:34390->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301934369 (60h47m5.024497419s) [noise=true , address=104.36.82.14:4221 , piece_id=TOSIDO6LOV6ZKD4GJHHQ5ZJVDT6NO3QYF7R5QUDIP3ROL7L5BYGA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301966701 (60h47m5.024529751s) [seg_pos={0 0}, upload=58285]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:55.535080726 (60h47m5.257643776s) [upload=58285]: Done uploading segments... Jul 27 00:22:55.535111805 (60h47m5.257674845s) [upload=58285]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235925155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXX] duration: 30.235081954s FAIL (2025-07-27 00:22:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235114888s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230520689s FAIL (2025-07-27 00:22:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== (60h45m45.622374518s) [noise=true , address=79.127.201.215:4347 , piece_id=RVPEXDTEBGLYUOFLW4SUWV3H3BRPWPCHSS5AOH6OHJFGHKPZLCUA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.899861905 (60h45m45.622424935s) [noise=true , address=104.36.82.53:4123 , piece_id=5T64IZEVUHWPT3RU2XWC7C75FJPMJSXHGXIQV6HGWK3RGJA7DGJA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900123364 (60h45m45.622686394s) [noise=true , address=79.127.220.97:4361 , piece_id=LF7SIQCDAWI3PU3OEFZEKIATCQLSZDPDDPKAVL5J4QNW6FTVOMXA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900159643 (60h45m45.622722673s) [noise=true , address=104.36.82.56:4248 , piece_id=LBR27M3RL6HWYUGBVWOLMUWA72UZWPVXRFN4IMGTNWQVILOIC35A, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900383953 (60h45m45.622946993s) [noise=true , address=77.90.31.17:4780 , piece_id=RVVEQEN4J2CQTJMQ57TXH6JWJFD56ICZDLJLOBYV57SRHUYUFXGQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900413790 (60h45m45.62297683s) [noise=true , address=79.127.205.245:4228 , piece_id=6AW7WYWTBS24AP7U7EQKX2CRPOPEKUMM7IBTBMUH3WMZ7SWG26TQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900619403 (60h45m45.623182433s) [noise=true , address=79.127.205.246:4260 , piece_id=4I3LUF3EF46HFSDSVWFYR3ECRI6AGAGISDGZ7WAIRNZ43TJXP47Q, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900851819 (60h45m45.623414859s) [noise=true , address=45.91.116.14:4194 , piece_id=GUGKMAK562MVIJY3RWZXEZ4YA2WJIMPD6R5YNYZOFE5A3REVGR7A, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901072160 (60h45m45.6236352s) [noise=true , address=77.90.31.13:4644 , piece_id=K4M3R5X54WGOQGOXV45GIVTTPBCOEA5VXBSSG5LGDUVRWG7UIJHA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901288794 (60h45m45.623851834s) [noise=true , address=104.36.82.46:4483 , piece_id=OLFZAE3K5IW673TRN3I3ICL7S54I5TXTYTLZAAPWMWMCHZAOI7RQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901516280 (60h45m45.62407932s) [noise=true , address=77.90.31.12:4603 , piece_id=UVPOPHV6OLJZ6EJA5VQJ24FWFWDXOZSP5TNLEWJSK2TYKVCIXVMA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901717444 (60h45m45.624280484s) [noise=true , address=79.127.205.233:4643 , piece_id=VAT2JO5OBUGJ4JYYNGLL55BNAAZX2YO3IJCCWTYEA23KVMOFIWTA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901951663 (60h45m45.624514703s) [noise=true , address=79.127.205.235:4684 , piece_id=PVTG7I2PZK4EN7OV3UWN3TFBI3GDGCFESUMEPGQJZRG456HFIXAA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.902162616 (60h45m45.624725656s) [noise=true , address=79.127.219.38:4003 , piece_id=4X2JXWC5DSM4H2P4WWEDTVJC6WVK44UX7Y5ZBZJQO3KVB2SMLVZA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.902350776 (60h45m45.624913816s) [noise=true , address=77.90.31.11:4575 , piece_id=TNTM2XXQJ7SCCN4P3FLJ4CHGD3MJH7LJETQTECVIS3ATT4IGRDYA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.908037276 (60h45m45.630600316s) [noise=true , address=104.36.82.54:4177 , piece_id=EPC6HYPTCB2G6SGIEILYWCZIHAKNJY4CZS3CS4LPY4SFXJJOXMDA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:37.007965269 (60h45m46.730528309s) [upload=58270]: Next returned nil segment Jul 27 00:21:37.007989225 (60h45m46.730552265s) [upload=58270]: Waiting for error group managing segments... Jul 27 00:21:37.292932447 (60h45m47.015495497s) [noise=true , address=79.127.205.245:4228 , piece_id=6AW7WYWTBS24AP7U7EQKX2CRPOPEKUMM7IBTBMUH3WMZ7SWG26TQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.302353503 (60h45m47.024916543s) [noise=true , address=79.127.205.246:4260 , piece_id=4I3LUF3EF46HFSDSVWFYR3ECRI6AGAGISDGZ7WAIRNZ43TJXP47Q, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.318647989 (60h45m47.041211029s) [noise=true , address=79.127.205.235:4684 , piece_id=PVTG7I2PZK4EN7OV3UWN3TFBI3GDGCFESUMEPGQJZRG456HFIXAA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.551562895 (60h45m47.274125935s) [noise=true , address=79.127.205.233:4643 , piece_id=VAT2JO5OBUGJ4JYYNGLL55BNAAZX2YO3IJCCWTYEA23KVMOFIWTA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:38.094470500 (60h45m47.81703354s) [noise=true , address=79.127.219.38:4003 , piece_id=4X2JXWC5DSM4H2P4WWEDTVJC6WVK44UX7Y5ZBZJQO3KVB2SMLVZA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:40.968569045 (60h45m50.691132075s) [noise=true , address=79.127.220.97:4361 , piece_id=LF7SIQCDAWI3PU3OEFZEKIATCQLSZDPDDPKAVL5J4QNW6FTVOMXA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:42.570418614 (60h45m52.292981654s) [noise=true , address=77.90.31.13:4644 , piece_id=K4M3R5X54WGOQGOXV45GIVTTPBCOEA5VXBSSG5LGDUVRWG7UIJHA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:49.883461706 (60h45m59.606024746s) [noise=true , address=45.91.116.14:4194 , piece_id=GUGKMAK562MVIJY3RWZXEZ4YA2WJIMPD6R5YNYZOFE5A3REVGR7A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:22:16.382783419 (60h46m26.105346449s) [noise=true , address=79.127.201.215:4347 , piece_id=RVPEXDTEBGLYUOFLW4SUWV3H3BRPWPCHSS5AOH6OHJFGHKPZLCUA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:22:34.796350709 (60h46m44.518913749s) [noise=true , address=77.90.31.11:4575 , piece_id=TNTM2XXQJ7SCCN4P3FLJ4CHGD3MJH7LJETQTECVIS3ATT4IGRDYA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 188.166.225.170:46826->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796387710 (60h46m44.51895074s) [noise=true , address=104.36.82.53:4123 , piece_id=5T64IZEVUHWPT3RU2XWC7C75FJPMJSXHGXIQV6HGWK3RGJA7DGJA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:104.36.82.53:4123): protocol: write tcp 188.166.225.170:60994->104.36.82.53:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796472282 (60h46m44.519035322s) [noise=true , address=104.36.82.54:4177 , piece_id=EPC6HYPTCB2G6SGIEILYWCZIHAKNJY4CZS3CS4LPY4SFXJJOXMDA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:39776->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796514873 (60h46m44.519077903s) [noise=true , address=104.36.82.41:4792 , piece_id=DO6QODRWE2DLNK4E4VQWMQ2NCUL4K5O2FZEL7TEBK7CR23L6RU3A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 188.166.225.170:40470->104.36.82.41:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796546034 (60h46m44.519109074s) [noise=true , address=104.36.82.46:4483 , piece_id=OLFZAE3K5IW673TRN3I3ICL7S54I5TXTYTLZAAPWMWMCHZAOI7RQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: write tcp 188.166.225.170:42064->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796566903 (60h46m44.519129943s) [noise=true , address=77.90.31.10:4535 , piece_id=VCQHGHHFZRPMKE3IUQV5VHN5BTH6WXEE7GMLTDO65XDYOYM56WXQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:60080->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796622439 (60h46m44.519185479s) [noise=true , address=77.90.31.12:4603 , piece_id=UVPOPHV6OLJZ6EJA5VQJ24FWFWDXOZSP5TNLEWJSK2TYKVCIXVMA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:22:34.796627359 (60h46m44.519190399s) [noise=true , address=104.36.82.56:4248 , piece_id=LBR27M3RL6HWYUGBVWOLMUWA72UZWPVXRFN4IMGTNWQVILOIC35A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: write tcp 188.166.225.170:45436->104.36.82.56:4248: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796683476 (60h46m44.519246517s) [noise=true , address=77.90.31.17:4780 , piece_id=RVVEQEN4J2CQTJMQ57TXH6JWJFD56ICZDLJLOBYV57SRHUYUFXGQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: read tcp 188.166.225.170:43882->77.90.31.17:4780: use of closed network connection; context deadline exceeded Jul 27 00:22:34.796704736 (60h46m44.519267776s) [seg_pos={0 0}, upload=58270]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:22:35.024803809 (60h46m44.747366849s) [upload=58270]: Done uploading segments... Jul 27 00:22:35.024832904 (60h46m44.747395984s) [upload=58270]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.23053658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X......X...XX..XX.X.XX...X...............XX......X...] duration: 49.701526399s FAIL (2025-07-27 00:22:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.701539616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.088074195s FAIL (2025-07-27 00:22:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.088104712s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X........] duration: 7.374887328s FAIL (2025-07-27 00:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.37490891s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XX] duration: 30.232568957s FAIL (2025-07-27 00:22:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5XA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; context deadline exceeded Jul 27 00:21:35.531548500 (60h45m45.25411154s) [seg_pos={0 0}, upload=58261]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:21:35.761627550 (60h45m45.48419059s) [upload=58261]: Done uploading segments... Jul 27 00:21:35.761652318 (60h45m45.484215358s) [upload=58261]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:21:36.634387963 (60h45m46.356950994s) [upload=58272]: Starting upload Jul 27 00:21:36.634444862 (60h45m46.357007902s) [upload=58272]: Done starting upload Jul 27 00:21:36.634605951 (60h45m46.357168991s) [upload=58272]: Uploading segments... Jul 27 00:21:36.634756889 (60h45m46.357319929s) [upload=58272]: Got next segment. Inline: false Jul 27 00:21:36.634822766 (60h45m46.357385806s) [upload=58272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:21:36.634829528 (60h45m46.357392568s) [upload=58272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:21:36.869871764 (60h45m46.592434804s) [seg_pos={0 0}, upload=58272]: Begin upload segment... Jul 27 00:21:36.869924796 (60h45m46.592487836s) [seg_pos={0 0}, upload=58272]: Done begin upload segment. Jul 27 00:21:36.869949984 (60h45m46.592513014s) [noise=true , address=77.90.31.15:4697 , piece_id=54FYZ7YHGMGG2FQ4RVKLOREX7VHCELNHZV4SGACPROVBOAJK4KNQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.869977898 (60h45m46.592540938s) [noise=true , address=104.36.82.7:4912 , piece_id=RHLYQUBM6TND3Y5DGYCVNCYGXSYBOSJROKFZQL7Z5D7HPJHQNE2Q, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870282761 (60h45m46.592845801s) [noise=true , address=45.91.116.10:4083 , piece_id=47RP7IMEY35X6WQSBEB2XFSFLNVTYGMEAEX4CVKRMFAR5W5EN4CQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870387622 (60h45m46.592950662s) [noise=true , address=45.91.116.14:4191 , piece_id=MFNRX6FLI4V6NHP6ICD2ID4QL2LCIKAAEDXJQZ23A6BXJI5JWFFA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870578717 (60h45m46.593141757s) [noise=true , address=45.91.116.13:4155 , piece_id=JKE3UKUHGFMKKTKHGJFELDPUZZANMVHYATNXP2CHELGRTKDRTUAQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870883170 (60h45m46.59344621s) [noise=true , address=45.91.116.8:4045 , piece_id=6HIVPBWMWCEOEOLQBB2RTGTIGLVBQBUIO73Y556OJVO4PSFJHMEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871235334 (60h45m46.593798374s) [noise=true , address=45.91.116.11:4117 , piece_id=XK4RWDOPR26DDNV2TLN45S5FGMZZC74BWF6JD7XSTOF7RBV3TZEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871552140 (60h45m46.59411518s) [noise=true , address=77.90.31.13:4626 , piece_id=U6TKZLNAD2GOMJLKD6DFLRBD4AVTX66J5DTRYIZ4NXZX7ISDIJ6Q, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871810995 (60h45m46.594374035s) [noise=true , address=45.91.116.9:4262 , piece_id=2B5LCT4Q44BHBVVAH53TZJ76IKCHLXI4EXZRMMOMLPK6K4D43PXA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872097092 (60h45m46.594660122s) [noise=true , address=45.91.116.6:4973 , piece_id=FGJGJZORKSAQ3JPLC2EN7I76CBWQ6MZW4N3LJITNJEGGSIXY3M5A, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872534589 (60h45m46.595097629s) [noise=true , address=104.36.82.15:4259 , piece_id=OUNXBRCJWK3FAPQ3RU6GXILFEACWBGVFE62OXABL4S5APDUAJ45A, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872853119 (60h45m46.595416159s) [noise=true , address=45.91.116.5:4299 , piece_id=YI53TB6KHTXYVK3LETQJ2LWUHHT2GBXFWCNXZYKHQ2THQAWSAGYQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873142062 (60h45m46.595705102s) [noise=true , address=77.90.31.12:4592 , piece_id=VDJ54S6BELCG7VFQ2IPBBEXVFWE23RM3N46PRK63W4JS3R2QOMWA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873507962 (60h45m46.596071002s) [noise=true , address=77.90.31.11:4554 , piece_id=YLDQHAAN3MU6PL64MTQ33TXFH2LAKL6IVOFCFJIVPUIQ6D5A7RUQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873766907 (60h45m46.596329947s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.874007598 (60h45m46.596570638s) [noise=true , address=104.36.82.9:4006 , piece_id=A23OFTCZU2RYHBKVBFCYXSTH5RPASRQAY7K53EIZ44GYJ56APHTA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:38.600943896 (60h45m48.323506936s) [upload=58272]: Next returned nil segment Jul 27 00:21:38.600959396 (60h45m48.323522436s) [upload=58272]: Waiting for error group managing segments... Jul 27 00:21:38.890679217 (60h45m48.613242257s) [noise=true , address=45.91.116.5:4299 , piece_id=YI53TB6KHTXYVK3LETQJ2LWUHHT2GBXFWCNXZYKHQ2THQAWSAGYQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.892006296 (60h45m48.614569336s) [noise=true , address=45.91.116.10:4083 , piece_id=47RP7IMEY35X6WQSBEB2XFSFLNVTYGMEAEX4CVKRMFAR5W5EN4CQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.896651554 (60h45m48.619214584s) [noise=true , address=45.91.116.9:4262 , piece_id=2B5LCT4Q44BHBVVAH53TZJ76IKCHLXI4EXZRMMOMLPK6K4D43PXA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.912248305 (60h45m48.634811355s) [noise=true , address=45.91.116.11:4117 , piece_id=XK4RWDOPR26DDNV2TLN45S5FGMZZC74BWF6JD7XSTOF7RBV3TZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.974286173 (60h45m48.696849213s) [noise=true , address=77.90.31.15:4697 , piece_id=54FYZ7YHGMGG2FQ4RVKLOREX7VHCELNHZV4SGACPROVBOAJK4KNQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.981026108 (60h45m48.703589148s) [noise=true , address=45.91.116.6:4973 , piece_id=FGJGJZORKSAQ3JPLC2EN7I76CBWQ6MZW4N3LJITNJEGGSIXY3M5A, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.984191114 (60h45m48.706754154s) [noise=true , address=45.91.116.13:4155 , piece_id=JKE3UKUHGFMKKTKHGJFELDPUZZANMVHYATNXP2CHELGRTKDRTUAQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.988692455 (60h45m48.711255495s) [noise=true , address=45.91.116.8:4045 , piece_id=6HIVPBWMWCEOEOLQBB2RTGTIGLVBQBUIO73Y556OJVO4PSFJHMEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:39.060708093 (60h45m48.783271133s) [noise=true , address=45.91.116.14:4191 , piece_id=MFNRX6FLI4V6NHP6ICD2ID4QL2LCIKAAEDXJQZ23A6BXJI5JWFFA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:40.276001385 (60h45m49.998564425s) [noise=true , address=77.90.31.13:4626 , piece_id=U6TKZLNAD2GOMJLKD6DFLRBD4AVTX66J5DTRYIZ4NXZX7ISDIJ6Q, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:40.495041585 (60h45m50.217604625s) [noise=true , address=77.90.31.12:4592 , piece_id=VDJ54S6BELCG7VFQ2IPBBEXVFWE23RM3N46PRK63W4JS3R2QOMWA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:22:05.994157902 (60h46m15.716720992s) [noise=true , address=104.36.82.9:4006 , piece_id=A23OFTCZU2RYHBKVBFCYXSTH5RPASRQAY7K53EIZ44GYJ56APHTA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:44256->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994185896 (60h46m15.716748996s) [noise=true , address=104.36.82.15:4259 , piece_id=OUNXBRCJWK3FAPQ3RU6GXILFEACWBGVFE62OXABL4S5APDUAJ45A, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:50648->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994263364 (60h46m15.716826444s) [noise=true , address=77.90.31.11:4554 , piece_id=YLDQHAAN3MU6PL64MTQ33TXFH2LAKL6IVOFCFJIVPUIQ6D5A7RUQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:40988->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994301777 (60h46m15.716864907s) [noise=true , address=104.36.82.7:4912 , piece_id=RHLYQUBM6TND3Y5DGYCVNCYGXSYBOSJROKFZQL7Z5D7HPJHQNE2Q, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:50860->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994355370 (60h46m15.71691841s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:22:05.994375438 (60h46m15.716938479s) [seg_pos={0 0}, upload=58272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:22:06.225001157 (60h46m15.947564197s) [upload=58272]: Done uploading segments... Jul 27 00:22:06.225029662 (60h46m15.947592702s) [upload=58272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.232580891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X......X...XX..XX.X.XX...X...............XX......X..] duration: 45.764965656s FAIL (2025-07-27 00:21:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.764977398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.833947582s FAIL (2025-07-27 00:21:44 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833976518s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXX] duration: 30.244238371s FAIL (2025-07-27 00:21:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244263169s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..X] duration: 30.233843873s FAIL (2025-07-27 00:21:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== QZ2FJOCMHCR2UWYE5NWAIYSLA, seg_pos={0 0}, upload=58245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:52248->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:01.592669710 (60h44m11.31523275s) [seg_pos={0 0}, upload=58245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:20:01.592779350 (60h44m11.31534239s) [upload=58245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:20:01.592784700 (60h44m11.31534774s) [upload=58245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:20:01.869390209 (60h44m11.591953259s) [upload=58245]: Done uploading segments... Jul 27 00:20:01.869476946 (60h44m11.592039996s) [upload=58245]: Upload finished. err: <nil> Jul 27 00:21:06.175009756 (60h45m15.897572816s) [upload=58261]: Starting upload Jul 27 00:21:06.175141508 (60h45m15.897704578s) [upload=58261]: Done starting upload Jul 27 00:21:06.175279542 (60h45m15.897842592s) [upload=58261]: Uploading segments... Jul 27 00:21:06.175540240 (60h45m15.89810329s) [upload=58261]: Got next segment. Inline: false Jul 27 00:21:06.175585557 (60h45m15.898148597s) [upload=58261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:21:06.175592631 (60h45m15.898155671s) [upload=58261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:21:06.406959225 (60h45m16.129522265s) [seg_pos={0 0}, upload=58261]: Begin upload segment... Jul 27 00:21:06.407011886 (60h45m16.129574926s) [seg_pos={0 0}, upload=58261]: Done begin upload segment. Jul 27 00:21:06.407032184 (60h45m16.129595224s) [noise=true , address=104.36.82.6:4874 , piece_id=W5JXWM6LI2DOGEGNSBQKUILZDRWM4XOQUIER22ZZYCUYZG4RT5XA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407079575 (60h45m16.129642615s) [noise=true , address=77.90.31.14:4663 , piece_id=GCUT2KJEF2IOQLZCSNXMMADKWQTLPT4CXTBCZLRIIHEL5UPYXUJA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407387894 (60h45m16.129950934s) [noise=true , address=45.91.116.8:4048 , piece_id=KB2R5TXDK4FRYNAQA7ZW6DWXCTJZJZV57AKVTLCYJ2JA5MKONAQQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407602665 (60h45m16.130165716s) [noise=true , address=45.91.116.9:4261 , piece_id=OJLT3LDSHFRZUWXIXGTJ7BQOJRWV4IBUXQN447U2CCNRT7GQH7QQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407648203 (60h45m16.130211243s) [noise=true , address=45.91.116.12:4225 , piece_id=DGLBA4WLVH6OVUOFCV4Q5ZZA7FXWYHCRQOONNDVA7UZOBJH7HAJA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407875728 (60h45m16.130438758s) [noise=true , address=45.91.116.7:4009 , piece_id=3BIFLZSWMA7UVCHTURVXTAZFOKXSTBVNW36EM2Z4OHYAA5ROMZTA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408041515 (60h45m16.130604555s) [noise=true , address=45.91.116.10:4083 , piece_id=G4SEWKI4SH6Z4RYNK4PBHFF4YZMXAZQFQP5ND4U75U2F6QGKRH2A, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408298907 (60h45m16.130861947s) [noise=true , address=77.90.31.21:4877 , piece_id=AWYRENUQ5CHGUMLIEH7PNQHGMOMA3IYBCIWF5VT72WOPDL5XKPFA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408400221 (60h45m16.130963262s) [noise=true , address=45.91.116.4:4940 , piece_id=J2MXG2QCOPIKHJ5UMOZ67HQZBE4YI35KREKJI4N2426DPP7DXTIQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408811458 (60h45m16.131374498s) [noise=true , address=45.91.116.12:4226 , piece_id=HL2QW36BRZKVX3QWYG3XVPHSHA4ZRMOYR7CL3TSTRFJJM5UDEKWA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409030918 (60h45m16.131593958s) [noise=true , address=77.90.31.14:4662 , piece_id=L733LLJ5PPYJLRGNTP4AZYSTK5G65OA7SEVE5W3JTQNWBNOGWUKA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409055926 (60h45m16.131618976s) [noise=true , address=45.91.116.7:4010 , piece_id=CCKW4QLBBKSN23LTUYZE2GXEWB5WVHP4UHHOMDG6KNX4W7OI2OAQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409310784 (60h45m16.131873824s) [noise=true , address=45.91.116.4:4937 , piece_id=BDMQX3HLQC5HOAFJPQMMLRRQUQOYD36B7JTIRXNADQC3TR5FNXVA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409532317 (60h45m16.132095357s) [noise=true , address=77.90.31.12:4589 , piece_id=AZUC5LNR572P37DKJVMENLQZMPNSJG6CXRRU5E4KFDKQPHYVJ5DA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409977360 (60h45m16.1325404s) [noise=true , address=104.36.82.14:4221 , piece_id=DOOTB4YCVPUSGVBEGGJINXGOXLFSOSTP2BCEMXNFTMCJOISTQRMA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.410214352 (60h45m16.132777393s) [noise=true , address=104.36.82.3:4767 , piece_id=OP62ICVPR4YCHG7YU7BLQMMOSUROOA4N5GWYWHFYTLDTSVSKBEKQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:08.132495033 (60h45m17.855058063s) [upload=58261]: Next returned nil segment Jul 27 00:21:08.132513439 (60h45m17.855076469s) [upload=58261]: Waiting for error group managing segments... Jul 27 00:21:08.423121589 (60h45m18.145684629s) [noise=true , address=45.91.116.4:4940 , piece_id=J2MXG2QCOPIKHJ5UMOZ67HQZBE4YI35KREKJI4N2426DPP7DXTIQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.438822710 (60h45m18.16138575s) [noise=true , address=45.91.116.12:4226 , piece_id=HL2QW36BRZKVX3QWYG3XVPHSHA4ZRMOYR7CL3TSTRFJJM5UDEKWA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.447263099 (60h45m18.169826139s) [noise=true , address=45.91.116.7:4009 , piece_id=3BIFLZSWMA7UVCHTURVXTAZFOKXSTBVNW36EM2Z4OHYAA5ROMZTA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.510942547 (60h45m18.233505587s) [noise=true , address=45.91.116.12:4225 , piece_id=DGLBA4WLVH6OVUOFCV4Q5ZZA7FXWYHCRQOONNDVA7UZOBJH7HAJA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.517049822 (60h45m18.239612862s) [noise=true , address=77.90.31.14:4662 , piece_id=L733LLJ5PPYJLRGNTP4AZYSTK5G65OA7SEVE5W3JTQNWBNOGWUKA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.519177914 (60h45m18.241740954s) [noise=true , address=77.90.31.14:4663 , piece_id=GCUT2KJEF2IOQLZCSNXMMADKWQTLPT4CXTBCZLRIIHEL5UPYXUJA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.521029547 (60h45m18.243592587s) [noise=true , address=45.91.116.9:4261 , piece_id=OJLT3LDSHFRZUWXIXGTJ7BQOJRWV4IBUXQN447U2CCNRT7GQH7QQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.531154458 (60h45m18.253717488s) [noise=true , address=45.91.116.10:4083 , piece_id=G4SEWKI4SH6Z4RYNK4PBHFF4YZMXAZQFQP5ND4U75U2F6QGKRH2A, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.531497534 (60h45m18.254060574s) [noise=true , address=45.91.116.7:4010 , piece_id=CCKW4QLBBKSN23LTUYZE2GXEWB5WVHP4UHHOMDG6KNX4W7OI2OAQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.573165027 (60h45m18.295728067s) [noise=true , address=45.91.116.8:4048 , piece_id=KB2R5TXDK4FRYNAQA7ZW6DWXCTJZJZV57AKVTLCYJ2JA5MKONAQQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.651239016 (60h45m18.373802056s) [noise=true , address=45.91.116.4:4937 , piece_id=BDMQX3HLQC5HOAFJPQMMLRRQUQOYD36B7JTIRXNADQC3TR5FNXVA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:09.854505436 (60h45m19.577068476s) [noise=true , address=77.90.31.12:4589 , piece_id=AZUC5LNR572P37DKJVMENLQZMPNSJG6CXRRU5E4KFDKQPHYVJ5DA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:35.531229941 (60h45m45.253792981s) [noise=true , address=77.90.31.21:4877 , piece_id=AWYRENUQ5CHGUMLIEH7PNQHGMOMA3IYBCIWF5VT72WOPDL5XKPFA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531306297 (60h45m45.253869337s) [noise=true , address=104.36.82.14:4221 , piece_id=DOOTB4YCVPUSGVBEGGJINXGOXLFSOSTP2BCEMXNFTMCJOISTQRMA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:46220->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531478837 (60h45m45.254041867s) [noise=true , address=104.36.82.3:4767 , piece_id=OP62ICVPR4YCHG7YU7BLQMMOSUROOA4N5GWYWHFYTLDTSVSKBEKQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:56604->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531513833 (60h45m45.254076874s) [noise=true , address=104.36.82.6:4874 , piece_id=W5JXWM6LI2DOGEGNSBQKUILZDRWM4XOQUIER22ZZYCUYZG4RT5XA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; context deadline exceeded Jul 27 00:21:35.531548500 (60h45m45.25411154s) [seg_pos={0 0}, upload=58261]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:21:35.761627550 (60h45m45.48419059s) [upload=58261]: Done uploading segments... Jul 27 00:21:35.761652318 (60h45m45.484215358s) [upload=58261]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233871567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238876021s FAIL (2025-07-27 00:21:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 424571561 (60h44m45.147134601s) [seg_pos={0 0}, upload=58255]: Done begin upload segment. Jul 27 00:20:35.424594505 (60h44m45.147157545s) [noise=true , address=104.36.82.7:4940 , piece_id=UKEUF6RRS4PWZXZWUQ7J3QCF3QBGUMGNJDV22CWTTKCXRIKG7ZVA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.424630664 (60h44m45.147193724s) [noise=true , address=77.90.31.11:4585 , piece_id=WKRFEUME7524PTG4IIONOF7ZXEGGVSWQYOEHOPT6OKSIDCJO6HGQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.424921541 (60h44m45.147484581s) [noise=true , address=79.127.220.101:4041 , piece_id=LTQQ23IJR7L4F5BHXXTXECVESMZVWTJVVPEXS6ZCAUHU6SIVPGEQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425312298 (60h44m45.147875338s) [noise=true , address=104.36.82.54:4174 , piece_id=O3NFUA65AIHQ5HGPFD6QIWJE2B67LCN7IF4KOJXPRUMZBM2AUGQQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425338829 (60h44m45.147901869s) [noise=true , address=45.91.116.9:4277 , piece_id=ITERKIQR4HPWVCQBABQD2WBLCQUMYONHI4BRQMI5LB2F3VBZQSAA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425619416 (60h44m45.148182446s) [noise=true , address=77.90.31.16:4743 , piece_id=RSPTFHDLZFQX32IIVTES3FQJXH4CVBHEJM763ESVM4P5GDDH7YWQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425623674 (60h44m45.148186714s) [noise=true , address=79.127.205.234:4618 , piece_id=PHHIBKQQ5VOOZQPPDYR5GHHBYA5LO6S46A33WVZAAXMC54SFRK4A, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425904220 (60h44m45.14846726s) [noise=true , address=77.90.31.18:4809 , piece_id=FR5PFVGFGDDG62ZYSXRUFZ3G5PAELFME62MSCDD5QNVXYPZ6G5KA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425975788 (60h44m45.148538828s) [noise=true , address=79.127.219.38:4018 , piece_id=7CLUGVYZRHVQXU3FPZPTGHXPATICXOC577HINQLQZXPBUYMGVWMA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426166362 (60h44m45.148729402s) [noise=true , address=64.49.13.138:4482 , piece_id=4FLURCOOV7NOQRRRQ4ETC4KLULNRDSLCUOQHDSGJXDER276237EQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426234893 (60h44m45.148797923s) [noise=true , address=77.90.31.15:4731 , piece_id=D23WPZMNCENQD3W23UMASXZP7O5XUV4NLQSPKVG2GRDETAN72ZSA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426363039 (60h44m45.148926079s) [noise=true , address=104.36.82.56:4262 , piece_id=XRW7RQNBUC3ANMKGMMH7NFDHRHIMX6M6QRXGANIEZSXCCH2NFPEQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426432210 (60h44m45.14899525s) [noise=true , address=77.90.31.13:4638 , piece_id=F53EJRYARYEUVA4WET2MK3737YE7H7EOPMECE7DCZWO2GD736R3Q, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426557420 (60h44m45.14912046s) [noise=true , address=104.36.82.47:4523 , piece_id=MJCYOBRC2QBBKSHCIC5MBPSFLLFE4VIPGBFJ25ZKRZ5BZY4A4GOQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426672901 (60h44m45.149235961s) [noise=true , address=79.127.205.246:4260 , piece_id=RCJ2F55PR5ETM5JUCCGJMOHBUQ2TA53SU645RJNQKD5CDSWQMKOA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426754527 (60h44m45.149317567s) [noise=true , address=104.36.82.46:4516 , piece_id=WQX2LHBHHU3O4LKULCG2ZSU77SDTQOXVODBWADVFRY2NZZGJ5QAQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.427151387 (60h44m45.149714427s) [noise=true , address=77.90.31.17:4790 , piece_id=JYEZQOD2BU7PQMW3F2D6WGO626K45OSWRUVZRA37EPZJCRNMOWWQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.427895691 (60h44m45.150458731s) [noise=true , address=79.127.201.215:4346 , piece_id=NPAKMSYWTBMKKJYHMWBRX7FHA4VYW6MFJM6OZAWA26W26KV6ULLA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:36.541327631 (60h44m46.263890671s) [upload=58255]: Next returned nil segment Jul 27 00:20:36.541345114 (60h44m46.263908144s) [upload=58255]: Waiting for error group managing segments... Jul 27 00:20:36.549751749 (60h44m46.272314779s) [noise=true , address=77.90.31.13:4638 , piece_id=F53EJRYARYEUVA4WET2MK3737YE7H7EOPMECE7DCZWO2GD736R3Q, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: storage node overloaded, request limit: 2000 Jul 27 00:20:36.840745377 (60h44m46.563308417s) [noise=true , address=79.127.205.234:4618 , piece_id=PHHIBKQQ5VOOZQPPDYR5GHHBYA5LO6S46A33WVZAAXMC54SFRK4A, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:36.852153805 (60h44m46.574716845s) [noise=true , address=79.127.205.246:4260 , piece_id=RCJ2F55PR5ETM5JUCCGJMOHBUQ2TA53SU645RJNQKD5CDSWQMKOA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.314951215 (60h44m47.037514255s) [noise=true , address=79.127.201.215:4346 , piece_id=NPAKMSYWTBMKKJYHMWBRX7FHA4VYW6MFJM6OZAWA26W26KV6ULLA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.336180925 (60h44m47.058743955s) [noise=true , address=79.127.219.38:4018 , piece_id=7CLUGVYZRHVQXU3FPZPTGHXPATICXOC577HINQLQZXPBUYMGVWMA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.352515446 (60h44m47.075078486s) [noise=true , address=77.90.31.18:4809 , piece_id=FR5PFVGFGDDG62ZYSXRUFZ3G5PAELFME62MSCDD5QNVXYPZ6G5KA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:38.143791747 (60h44m47.866354787s) [noise=true , address=77.90.31.16:4743 , piece_id=RSPTFHDLZFQX32IIVTES3FQJXH4CVBHEJM763ESVM4P5GDDH7YWQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:39.152402764 (60h44m48.874965804s) [noise=true , address=64.49.13.138:4482 , piece_id=4FLURCOOV7NOQRRRQ4ETC4KLULNRDSLCUOQHDSGJXDER276237EQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:40.836663303 (60h44m50.559226333s) [noise=true , address=45.91.116.9:4277 , piece_id=ITERKIQR4HPWVCQBABQD2WBLCQUMYONHI4BRQMI5LB2F3VBZQSAA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:43.987867722 (60h44m53.710430782s) [noise=true , address=79.127.220.101:4041 , piece_id=LTQQ23IJR7L4F5BHXXTXECVESMZVWTJVVPEXS6ZCAUHU6SIVPGEQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:57.821042376 (60h45m7.543605416s) [noise=true , address=77.90.31.15:4731 , piece_id=D23WPZMNCENQD3W23UMASXZP7O5XUV4NLQSPKVG2GRDETAN72ZSA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:21:34.333503139 (60h45m44.056066179s) [noise=true , address=77.90.31.11:4585 , piece_id=WKRFEUME7524PTG4IIONOF7ZXEGGVSWQYOEHOPT6OKSIDCJO6HGQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333545630 (60h45m44.05610867s) [noise=true , address=104.36.82.46:4516 , piece_id=WQX2LHBHHU3O4LKULCG2ZSU77SDTQOXVODBWADVFRY2NZZGJ5QAQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: rpc: dial tcp 104.36.82.46:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333577962 (60h45m44.056141002s) [noise=true , address=104.36.82.56:4262 , piece_id=XRW7RQNBUC3ANMKGMMH7NFDHRHIMX6M6QRXGANIEZSXCCH2NFPEQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333607038 (60h45m44.056170078s) [noise=true , address=104.36.82.47:4523 , piece_id=MJCYOBRC2QBBKSHCIC5MBPSFLLFE4VIPGBFJ25ZKRZ5BZY4A4GOQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 188.166.225.170:56130->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333647405 (60h45m44.056210446s) [noise=true , address=77.90.31.17:4790 , piece_id=JYEZQOD2BU7PQMW3F2D6WGO626K45OSWRUVZRA37EPZJCRNMOWWQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:21:34.333651112 (60h45m44.056214152s) [noise=true , address=104.36.82.54:4174 , piece_id=O3NFUA65AIHQ5HGPFD6QIWJE2B67LCN7IF4KOJXPRUMZBM2AUGQQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333658977 (60h45m44.056222017s) [noise=true , address=104.36.82.7:4940 , piece_id=UKEUF6RRS4PWZXZWUQ7J3QCF3QBGUMGNJDV22CWTTKCXRIKG7ZVA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:104.36.82.7:4940): protocol: rpc: dial tcp 104.36.82.7:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333672162 (60h45m44.056235202s) [seg_pos={0 0}, upload=58255]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:21:34.570295584 (60h45m44.292858624s) [upload=58255]: Done uploading segments... Jul 27 00:21:34.570325080 (60h45m44.29288812s) [upload=58255]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.238888265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.826511986s FAIL (2025-07-27 00:21:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.826537125s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.......] duration: 7.463890186s FAIL (2025-07-27 00:21:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.463907958s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X..X.....X......X...XX..XX.X.XX...X...............XX......X.] duration: 56.623183294s FAIL (2025-07-27 00:20:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.623195779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X......] duration: 6.369788097s FAIL (2025-07-27 00:20:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.369810501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23523877s FAIL (2025-07-27 00:20:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 7293336s) [noise=true , address=79.127.219.48:4074 , piece_id=LEZQ4DDHMZVGUJR5IDAL2QUPYIYADFDA3DKCGGOEQBZY3LRYH4OA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.954813295 (60h43m44.677376325s) [noise=true , address=77.90.31.11:4575 , piece_id=UBD3K5OFOLEGY52A7Q47G3ZAJ3J56RDMUBX7SUD4HOQO5TP5AE4A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955085847 (60h43m44.677648887s) [noise=true , address=104.36.82.46:4493 , piece_id=PDNBKD2BHCATEJNK52V2J5WZHW3PVJCP2VUXS3ZIJ6CPPU3H4TAA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955335805 (60h43m44.677898845s) [noise=true , address=64.49.13.137:4502 , piece_id=OQWQP3TEDSZ2K3U2AJRL462WZEXBKBZBM2MJUFKKTFBETCNW4IDQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955583218 (60h43m44.678146258s) [noise=true , address=79.127.219.38:4006 , piece_id=CDHYTG33QZLWVGX6ISIZUHM6XR3QFCWSOOPTZUVGLL43BDMWI3IA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955821484 (60h43m44.678384524s) [noise=true , address=104.36.82.54:4163 , piece_id=HN35V7VNNRR2TGIDM5PLXPDMKLR2ZFYO2TZI4NDSFBLFWR7J5AAQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956036766 (60h43m44.678599806s) [noise=true , address=77.90.31.18:4823 , piece_id=FZLULHRAMEYKJ254U5XWAUWV7RJFHSVEX4DLVJ3CGJDSUCNFEJKA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956194448 (60h43m44.678757478s) [noise=true , address=79.127.205.245:4245 , piece_id=CJM7ISA7PFNHW2D7XRGRKVDZYTJC7IDH5DDX3BAANI7ZNHXRSU4Q, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956442703 (60h43m44.679005733s) [noise=true , address=104.36.82.47:4537 , piece_id=USZNOO2ROXDEX4ZOLHL27OBJKZFUYY2WDEA6BN44UW4WNIPE4SGQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956666812 (60h43m44.679229852s) [noise=true , address=45.91.116.8:4062 , piece_id=LN7P5JGQMFS5LOSDCF7OO4RPODA76C2ZCFBTWAVMDTU3GRN54KRQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956709563 (60h43m44.679272603s) [noise=true , address=79.127.205.246:4282 , piece_id=T2AIQLDZUY27WAHBUNBPBN3I2DPY3UVWBT3BLCOWC5NDDNBZYPSQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956923593 (60h43m44.679486633s) [noise=true , address=104.36.82.11:4116 , piece_id=V2722NTPPIN5C4CCXEEXRKNDUAKTVSC7DJF5N4SDSGWNJCPM26JA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957114127 (60h43m44.679677168s) [noise=true , address=104.36.82.55:4202 , piece_id=4QG5MNIRYMX5DYQ72D2CH5ZTZDUO2NTIEDTNLYJ6ZJJX6NDWRDFQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957265508 (60h43m44.679828548s) [noise=true , address=79.127.201.214:4321 , piece_id=M62AFBCW435H6PQEWEALBJKZU6NGQPETTDI6Y6KGE5JJECOLW77Q, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957479977 (60h43m44.680043017s) [noise=true , address=79.127.220.97:4361 , piece_id=O52ZWST27VEH2JQYT5BGK6ZRPDZI6ZB4EJDQNHG2OXZU5EM4OPWQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957562455 (60h43m44.680125485s) [noise=true , address=104.36.82.52:4096 , piece_id=IQRCWGOOFGD3TFDPE6JGEJQZTY372EGUOWFXUAYB265G2EROTB5A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957750064 (60h43m44.680313094s) [noise=true , address=104.36.82.14:4244 , piece_id=VM5UEPCKLGDUF22ZSFHVI3OMJEYKI4LCBJIQYSCSEI6Z2XG4D77A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:36.073032685 (60h43m45.795595725s) [upload=58240]: Next returned nil segment Jul 27 00:19:36.073050499 (60h43m45.795613529s) [upload=58240]: Waiting for error group managing segments... Jul 27 00:19:36.396390456 (60h43m46.118953496s) [noise=true , address=79.127.205.245:4245 , piece_id=CJM7ISA7PFNHW2D7XRGRKVDZYTJC7IDH5DDX3BAANI7ZNHXRSU4Q, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.481879673 (60h43m46.204442703s) [noise=true , address=64.49.13.137:4502 , piece_id=OQWQP3TEDSZ2K3U2AJRL462WZEXBKBZBM2MJUFKKTFBETCNW4IDQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.765119535 (60h43m46.487682575s) [noise=true , address=79.127.220.97:4361 , piece_id=O52ZWST27VEH2JQYT5BGK6ZRPDZI6ZB4EJDQNHG2OXZU5EM4OPWQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.859649860 (60h43m46.58221289s) [noise=true , address=79.127.205.246:4282 , piece_id=T2AIQLDZUY27WAHBUNBPBN3I2DPY3UVWBT3BLCOWC5NDDNBZYPSQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.947368111 (60h43m46.669931152s) [noise=true , address=79.127.219.48:4074 , piece_id=LEZQ4DDHMZVGUJR5IDAL2QUPYIYADFDA3DKCGGOEQBZY3LRYH4OA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.958322321 (60h43m46.680885361s) [noise=true , address=79.127.219.38:4006 , piece_id=CDHYTG33QZLWVGX6ISIZUHM6XR3QFCWSOOPTZUVGLL43BDMWI3IA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:37.085849138 (60h43m46.808412178s) [noise=true , address=77.90.31.18:4823 , piece_id=FZLULHRAMEYKJ254U5XWAUWV7RJFHSVEX4DLVJ3CGJDSUCNFEJKA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:38.455975221 (60h43m48.178538251s) [noise=true , address=79.127.201.214:4321 , piece_id=M62AFBCW435H6PQEWEALBJKZU6NGQPETTDI6Y6KGE5JJECOLW77Q, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:57.616809053 (60h44m7.339372093s) [noise=true , address=45.91.116.8:4062 , piece_id=LN7P5JGQMFS5LOSDCF7OO4RPODA76C2ZCFBTWAVMDTU3GRN54KRQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:20:33.848344606 (60h44m43.570907646s) [noise=true , address=104.36.82.46:4493 , piece_id=PDNBKD2BHCATEJNK52V2J5WZHW3PVJCP2VUXS3ZIJ6CPPU3H4TAA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848424269 (60h44m43.570987309s) [noise=true , address=104.36.82.54:4163 , piece_id=HN35V7VNNRR2TGIDM5PLXPDMKLR2ZFYO2TZI4NDSFBLFWR7J5AAQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848478953 (60h44m43.571042004s) [noise=true , address=104.36.82.52:4096 , piece_id=IQRCWGOOFGD3TFDPE6JGEJQZTY372EGUOWFXUAYB265G2EROTB5A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: write tcp 188.166.225.170:50222->104.36.82.52:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848592030 (60h44m43.57115507s) [noise=true , address=77.90.31.17:4786 , piece_id=RJ5FD7P7HCVP3SQZKWARZCFZFTNCCD25DNUXNHCVB2CQOUWSRO6A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: rpc: dial tcp 77.90.31.17:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848641575 (60h44m43.571204615s) [noise=true , address=77.90.31.11:4575 , piece_id=UBD3K5OFOLEGY52A7Q47G3ZAJ3J56RDMUBX7SUD4HOQO5TP5AE4A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848687993 (60h44m43.571251033s) [noise=true , address=104.36.82.11:4116 , piece_id=V2722NTPPIN5C4CCXEEXRKNDUAKTVSC7DJF5N4SDSGWNJCPM26JA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:104.36.82.11:4116): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:33.848698293 (60h44m43.571261333s) [noise=true , address=104.36.82.47:4537 , piece_id=USZNOO2ROXDEX4ZOLHL27OBJKZFUYY2WDEA6BN44UW4WNIPE4SGQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848757005 (60h44m43.571320045s) [noise=true , address=104.36.82.55:4202 , piece_id=4QG5MNIRYMX5DYQ72D2CH5ZTZDUO2NTIEDTNLYJ6ZJJX6NDWRDFQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848772515 (60h44m43.571335566s) [noise=true , address=104.36.82.14:4244 , piece_id=VM5UEPCKLGDUF22ZSFHVI3OMJEYKI4LCBJIQYSCSEI6Z2XG4D77A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: write tcp 188.166.225.170:53038->104.36.82.14:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848793856 (60h44m43.571356896s) [seg_pos={0 0}, upload=58240]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:20:34.081476006 (60h44m43.804039046s) [upload=58240]: Done uploading segments... Jul 27 00:20:34.081502797 (60h44m43.804065837s) [upload=58240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.23525465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXX] duration: 30.243351076s FAIL (2025-07-27 00:20:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.243363519s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.110352396s FAIL (2025-07-27 00:20:20 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.110378826s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.....] duration: 7.773077094s FAIL (2025-07-27 00:20:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.773097984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..] duration: 6.812501964s FAIL (2025-07-27 00:20:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:18:24.334097790 (60h42m34.05666082s) [upload=58224]: Starting upload Jul 27 00:18:24.334154699 (60h42m34.056717779s) [upload=58224]: Done starting upload Jul 27 00:18:24.334237908 (60h42m34.056800938s) [upload=58224]: Uploading segments... Jul 27 00:18:24.334437640 (60h42m34.05700067s) [upload=58224]: Got next segment. Inline: false Jul 27 00:18:24.334483638 (60h42m34.057046679s) [upload=58224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:24.334488578 (60h42m34.057051618s) [upload=58224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:24.564881187 (60h42m34.287444227s) [seg_pos={0 0}, upload=58224]: Begin upload segment... Jul 27 00:18:24.564929989 (60h42m34.28749303s) [seg_pos={0 0}, upload=58224]: Done begin upload segment. Jul 27 00:18:24.564954016 (60h42m34.287517056s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.564998090 (60h42m34.28756112s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565311169 (60h42m34.287874209s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565454935 (60h42m34.288017965s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565559765 (60h42m34.288122805s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565729580 (60h42m34.28829262s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565840673 (60h42m34.288403713s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566026167 (60h42m34.288589207s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566132010 (60h42m34.28869505s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566382689 (60h42m34.288945719s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566620755 (60h42m34.289183795s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566845164 (60h42m34.289408204s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567029938 (60h42m34.289592978s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567212567 (60h42m34.289775607s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567393123 (60h42m34.289956163s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567580712 (60h42m34.290143742s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:26.259818721 (60h42m35.982381751s) [upload=58224]: Next returned nil segment Jul 27 00:18:26.259835624 (60h42m35.982398654s) [upload=58224]: Waiting for error group managing segments... Jul 27 00:18:26.584579627 (60h42m36.307142667s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592038658 (60h42m36.314601688s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592821997 (60h42m36.315385038s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.662347510 (60h42m36.38491055s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.679168664 (60h42m36.401731704s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.700554562 (60h42m36.423117592s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.736485559 (60h42m36.459048589s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:27.128265201 (60h42m36.850828241s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.113985780 (60h42m37.83654882s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.206936645 (60h42m37.929499685s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:53.685476775 (60h43m3.408039815s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685723547 (60h43m3.408286577s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:59780->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685745219 (60h43m3.408308249s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685781728 (60h43m3.408344768s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685849078 (60h43m3.408412118s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685877802 (60h43m3.408440842s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685897851 (60h43m3.408460891s) [seg_pos={0 0}, upload=58224]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:53.911235189 (60h43m3.633798229s) [upload=58224]: Done uploading segments... Jul 27 00:18:53.911264825 (60h43m3.633827865s) [upload=58224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.812526923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XX..X.....X......X...XX..XX.X.XX...X...............XX......X] duration: 1m0.001500679s FAIL (2025-07-27 00:20:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001526397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XX] duration: 30.231894723s FAIL (2025-07-27 00:19:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.231912506s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX.] duration: 7.163360941s FAIL (2025-07-27 00:19:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:18:24.334097790 (60h42m34.05666082s) [upload=58224]: Starting upload Jul 27 00:18:24.334154699 (60h42m34.056717779s) [upload=58224]: Done starting upload Jul 27 00:18:24.334237908 (60h42m34.056800938s) [upload=58224]: Uploading segments... Jul 27 00:18:24.334437640 (60h42m34.05700067s) [upload=58224]: Got next segment. Inline: false Jul 27 00:18:24.334483638 (60h42m34.057046679s) [upload=58224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:24.334488578 (60h42m34.057051618s) [upload=58224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:24.564881187 (60h42m34.287444227s) [seg_pos={0 0}, upload=58224]: Begin upload segment... Jul 27 00:18:24.564929989 (60h42m34.28749303s) [seg_pos={0 0}, upload=58224]: Done begin upload segment. Jul 27 00:18:24.564954016 (60h42m34.287517056s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.564998090 (60h42m34.28756112s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565311169 (60h42m34.287874209s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565454935 (60h42m34.288017965s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565559765 (60h42m34.288122805s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565729580 (60h42m34.28829262s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565840673 (60h42m34.288403713s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566026167 (60h42m34.288589207s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566132010 (60h42m34.28869505s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566382689 (60h42m34.288945719s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566620755 (60h42m34.289183795s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566845164 (60h42m34.289408204s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567029938 (60h42m34.289592978s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567212567 (60h42m34.289775607s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567393123 (60h42m34.289956163s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567580712 (60h42m34.290143742s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:26.259818721 (60h42m35.982381751s) [upload=58224]: Next returned nil segment Jul 27 00:18:26.259835624 (60h42m35.982398654s) [upload=58224]: Waiting for error group managing segments... Jul 27 00:18:26.584579627 (60h42m36.307142667s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592038658 (60h42m36.314601688s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592821997 (60h42m36.315385038s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.662347510 (60h42m36.38491055s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.679168664 (60h42m36.401731704s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.700554562 (60h42m36.423117592s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.736485559 (60h42m36.459048589s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:27.128265201 (60h42m36.850828241s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.113985780 (60h42m37.83654882s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.206936645 (60h42m37.929499685s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:53.685476775 (60h43m3.408039815s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685723547 (60h43m3.408286577s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:59780->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685745219 (60h43m3.408308249s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685781728 (60h43m3.408344768s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685849078 (60h43m3.408412118s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685877802 (60h43m3.408440842s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685897851 (60h43m3.408460891s) [seg_pos={0 0}, upload=58224]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:53.911235189 (60h43m3.633798229s) [upload=58224]: Done uploading segments... Jul 27 00:18:53.911264825 (60h43m3.633827865s) [upload=58224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.163405718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234351135s FAIL (2025-07-27 00:19:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece_id=LUEEX3QN6AM2FT75GANCNOS6PQGZCZWKH4F3VCH24LPNFT56W4CA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.481894408 (60h42m44.204457448s) [noise=true , address=104.36.82.46:4501 , piece_id=GJYUQA7AFKYED4N7H2ONXWTD4SQ7OLQTTVVHQDRQ2OV7HPP37QWQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482222586 (60h42m44.204785626s) [noise=true , address=79.127.205.234:4607 , piece_id=DWAZDYUI74PLSMOZKHJ3MFRC65NFWIYRIACVPKLZZRLXVBKSYJRA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482581452 (60h42m44.205144482s) [noise=true , address=77.90.31.19:4871 , piece_id=4XEVC563OQI7YCCQ2JRDR66NKM5RORT2A2ZPOWD65WZJ5QSEC2ZQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482847722 (60h42m44.205410762s) [noise=true , address=77.90.31.17:4778 , piece_id=QIYIIBCZGQPNNOVHC7ZNF76VAACNJGMXTCO5EQUJZE4DNELQH6IQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483105084 (60h42m44.205668124s) [noise=true , address=104.36.82.55:4194 , piece_id=HNEE5JZ6BZ5JHGROQSSKDSBIGUMRLGDVLJJ6GSYA4KKIMBRVLKQQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483358819 (60h42m44.205921859s) [noise=true , address=64.49.13.138:4488 , piece_id=TE7I4CJMA767HHH5YKWMLJOE5LS5M32NGI76X62WQBO2G5NHGM4Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483622043 (60h42m44.206185083s) [noise=true , address=77.90.31.16:4751 , piece_id=LJB7N2VOHYA43XZP32I4RPHRI4G63IWF33JVFBIQY56YFYIEAU2A, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483821244 (60h42m44.206384284s) [noise=true , address=77.90.31.18:4817 , piece_id=LAFPCQDUFJCZQQRGIIIMGKT3RY527BYIN55QPS2KGF4ADAWLI3UA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484083887 (60h42m44.206646927s) [noise=true , address=104.36.82.54:4190 , piece_id=SPKGL26ZFVI6VWJQIGBOG3BZZ2Z4RTPHUGLDQUIKZRWTEFRG463Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484286775 (60h42m44.206849815s) [noise=true , address=45.91.116.9:4277 , piece_id=KO52SDQPK4MCQD7XAF7Q5ZCIKAGBVRGFPWKXZMW45C4TDWDIM56Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484480996 (60h42m44.207044036s) [noise=true , address=104.36.82.47:4546 , piece_id=44H6TXDATOOSKBWLNRBW6D77CUT4RBDONMXI34UOEAUGLQCBBJTQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484665229 (60h42m44.207228259s) [noise=true , address=79.127.219.38:4001 , piece_id=KMTLQ2CLCS6ZF66SF7JSPUPPIWZVQFXYPCKDS3KIDDMPVJFLDUEA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484863058 (60h42m44.207426098s) [noise=true , address=104.36.82.52:4119 , piece_id=27ZPC7G2IMJJ6GQWD3WZGE7RFZKPXKOSPUE7HO2SQHDW3KIBSR5Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485056027 (60h42m44.207619057s) [noise=true , address=79.127.205.246:4283 , piece_id=SVLSGY6XYJFPIBQLBHBQFZTCKLAINYZABS5PKGCL7TC3CJM7Q6RQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485242083 (60h42m44.207805123s) [noise=true , address=77.90.31.11:4576 , piece_id=7ZW7TOSVB5BPFZT4HGNCBBRCTESX5GNM5V5ULBLYU4MYUFAVZEJQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485441644 (60h42m44.208004685s) [noise=true , address=104.36.82.53:4147 , piece_id=COCCHCNFRMA425SNFZLKTZNLCXYEEATE3IWP4KERGKJ72VL47BNA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485668378 (60h42m44.208231408s) [noise=true , address=77.90.31.12:4611 , piece_id=JTFTDCB2KJ3WJ75XDXUYGEN55QL4WKCKPG2TW5TE7FGT3TLQFO6A, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:35.510716284 (60h42m45.233279324s) [noise=true , address=77.90.31.19:4871 , piece_id=4XEVC563OQI7YCCQ2JRDR66NKM5RORT2A2ZPOWD65WZJ5QSEC2ZQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 27 00:18:35.605175292 (60h42m45.327738322s) [upload=58226]: Next returned nil segment Jul 27 00:18:35.605190971 (60h42m45.327754011s) [upload=58226]: Waiting for error group managing segments... Jul 27 00:18:35.887562934 (60h42m45.610125974s) [noise=true , address=79.127.205.246:4283 , piece_id=SVLSGY6XYJFPIBQLBHBQFZTCKLAINYZABS5PKGCL7TC3CJM7Q6RQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:35.895500381 (60h42m45.618063421s) [noise=true , address=79.127.205.233:4650 , piece_id=LUEEX3QN6AM2FT75GANCNOS6PQGZCZWKH4F3VCH24LPNFT56W4CA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:35.904784985 (60h42m45.627348015s) [noise=true , address=79.127.205.234:4607 , piece_id=DWAZDYUI74PLSMOZKHJ3MFRC65NFWIYRIACVPKLZZRLXVBKSYJRA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.392178315 (60h42m46.114741355s) [noise=true , address=77.90.31.18:4817 , piece_id=LAFPCQDUFJCZQQRGIIIMGKT3RY527BYIN55QPS2KGF4ADAWLI3UA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.436969825 (60h42m46.159532865s) [noise=true , address=64.49.13.138:4488 , piece_id=TE7I4CJMA767HHH5YKWMLJOE5LS5M32NGI76X62WQBO2G5NHGM4Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.569996585 (60h42m46.292559615s) [noise=true , address=77.90.31.16:4751 , piece_id=LJB7N2VOHYA43XZP32I4RPHRI4G63IWF33JVFBIQY56YFYIEAU2A, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:37.028571727 (60h42m46.751134767s) [noise=true , address=79.127.219.38:4001 , piece_id=KMTLQ2CLCS6ZF66SF7JSPUPPIWZVQFXYPCKDS3KIDDMPVJFLDUEA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:46.549073710 (60h42m56.27163675s) [noise=true , address=77.90.31.11:4576 , piece_id=7ZW7TOSVB5BPFZT4HGNCBBRCTESX5GNM5V5ULBLYU4MYUFAVZEJQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:52.538901001 (60h43m2.261464041s) [noise=true , address=77.90.31.12:4611 , piece_id=JTFTDCB2KJ3WJ75XDXUYGEN55QL4WKCKPG2TW5TE7FGT3TLQFO6A, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:52.716311530 (60h43m2.43887457s) [noise=true , address=45.91.116.9:4277 , piece_id=KO52SDQPK4MCQD7XAF7Q5ZCIKAGBVRGFPWKXZMW45C4TDWDIM56Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:19:33.377643345 (60h43m43.100206385s) [noise=true , address=77.90.31.17:4778 , piece_id=QIYIIBCZGQPNNOVHC7ZNF76VAACNJGMXTCO5EQUJZE4DNELQH6IQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: write tcp 188.166.225.170:45538->77.90.31.17:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377710294 (60h43m43.100273344s) [noise=true , address=104.36.82.53:4147 , piece_id=COCCHCNFRMA425SNFZLKTZNLCXYEEATE3IWP4KERGKJ72VL47BNA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377770599 (60h43m43.100333639s) [noise=true , address=104.36.82.52:4119 , piece_id=27ZPC7G2IMJJ6GQWD3WZGE7RFZKPXKOSPUE7HO2SQHDW3KIBSR5Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: write tcp 188.166.225.170:48056->104.36.82.52:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377842056 (60h43m43.100405096s) [noise=true , address=104.36.82.46:4501 , piece_id=GJYUQA7AFKYED4N7H2ONXWTD4SQ7OLQTTVVHQDRQ2OV7HPP37QWQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377904786 (60h43m43.100467826s) [noise=true , address=104.36.82.55:4194 , piece_id=HNEE5JZ6BZ5JHGROQSSKDSBIGUMRLGDVLJJ6GSYA4KKIMBRVLKQQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 188.166.225.170:37680->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377910507 (60h43m43.100473537s) [noise=true , address=104.36.82.54:4190 , piece_id=SPKGL26ZFVI6VWJQIGBOG3BZZ2Z4RTPHUGLDQUIKZRWTEFRG463Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: write tcp 188.166.225.170:52512->104.36.82.54:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377921829 (60h43m43.100484869s) [noise=true , address=104.36.82.47:4546 , piece_id=44H6TXDATOOSKBWLNRBW6D77CUT4RBDONMXI34UOEAUGLQCBBJTQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377934783 (60h43m43.100497823s) [seg_pos={0 0}, upload=58226]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:19:33.610061660 (60h43m43.332624701s) [upload=58226]: Done uploading segments... Jul 27 00:19:33.610086097 (60h43m43.332649137s) [upload=58226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234363198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.X] duration: 30.241082604s FAIL (2025-07-27 00:19:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.241107782s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.866204039s FAIL (2025-07-27 00:19:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.866227585s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)