Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.907652455s FAIL (2025-07-26 23:28:05 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.907675309s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23096054s FAIL (2025-07-26 23:27:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== IA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.294275546 (59h51m2.016838576s) [noise=true , address=104.36.82.22:4510 , piece_id=CVVXR3BL3V2QGZC2M3CKDMDJWWNYHQ5GZ4GI2EMYTQQXZMGGEPDA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.294523941 (59h51m2.017086981s) [noise=true , address=104.36.82.53:4127 , piece_id=Y7AOQ6E64QQLOVIO4HQRWAJRMOFRFPRTYPTO4O3PBYNC2ZGNHZJQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295386402 (59h51m2.017949442s) [noise=true , address=104.36.82.46:4490 , piece_id=F5SINJ2OCMIH7SMEG5IOKPLGNQDGOJ3JWVBEXQ5ZV4N2FILG5R7A, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295669744 (59h51m2.018232784s) [noise=true , address=79.127.219.47:4048 , piece_id=ZNYUAXIUKUQRWQT536X2PEKHAORKGUNUWTSDGMP3HPAOS2LGYBLQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295942115 (59h51m2.018505155s) [noise=true , address=156.146.43.225:4453 , piece_id=7FM3YH327FRUDPLPJM35UKUJMFQ3UPP56OSXS7MNW6B3GKBRXMWA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296407606 (59h51m2.018970636s) [noise=true , address=104.36.82.21:4503 , piece_id=JROIOE56GPG2AJT7HWSZUMMZFWJ7NFB6LPB3Z3JJZBO5CC2AC3IQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296637325 (59h51m2.019200365s) [noise=true , address=104.36.82.52:4097 , piece_id=J5VOVTTNSU2XPBTCEAPAAJOFTPM364R7CCHRVC52PAJGSKEJNH3Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296832348 (59h51m2.019395388s) [noise=true , address=79.127.205.246:4272 , piece_id=5B7LE4Z25OUGZGH7WQBF5KV2G2UP3QKAB5TTUUKNY2FEAQ62KTOQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297034936 (59h51m2.019597976s) [noise=true , address=79.127.205.235:4682 , piece_id=SV5NQJWIUQMAMSO264FCRZ6N7BD2GJXDSXYURGLU23WOBY2TUK3A, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297261900 (59h51m2.01982494s) [noise=true , address=79.127.205.234:4619 , piece_id=3BPZ3VHTHETHWDC4XIE42QFAUKAS47T2MLYRZ3XGJ5JLTAYFXLAA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297468967 (59h51m2.020032007s) [noise=true , address=104.36.82.54:4157 , piece_id=HJM5A37YBHG4O2N3W6KWBNFZWHDHOXFZ7YI3FLUXGWNV3ALX723Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297713734 (59h51m2.020276774s) [noise=true , address=79.127.205.245:4221 , piece_id=QW4VM7F7MDOIDFQT32R5REAPLI7ZZTS4HZNFG3BEGX4BTHN6XOOQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298040980 (59h51m2.02060402s) [noise=true , address=104.36.82.47:4523 , piece_id=UDBVQ2UII4ZJGW6HPMXG4UMPXW46KVN7YHFIQ5UUXGPUNQOK6O7Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298270851 (59h51m2.020833891s) [noise=true , address=79.127.220.101:4048 , piece_id=EY4DOTTE72JJMXUWA7MVQ2Z6ENRVPLSMVGCXRON7FMI5FDQV5TBA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298378517 (59h51m2.020941557s) [noise=true , address=104.36.82.45:4924 , piece_id=LFONCFVCAKPEQ7PWAU5PXJ4C7FRFENACAV352WEMWRYYDETBUACA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298636640 (59h51m2.02119968s) [noise=true , address=79.127.219.38:4017 , piece_id=J66T5WVTOZV7MDOEE7ICJNQNPCP3EXRZWENUPQWRNR5S7RE5QK5Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:53.383003392 (59h51m3.105566432s) [upload=57424]: Next returned nil segment Jul 26 23:26:53.383020544 (59h51m3.105583585s) [upload=57424]: Waiting for error group managing segments... Jul 26 23:26:53.686724482 (59h51m3.409287512s) [noise=true , address=79.127.205.235:4682 , piece_id=SV5NQJWIUQMAMSO264FCRZ6N7BD2GJXDSXYURGLU23WOBY2TUK3A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.699720738 (59h51m3.422283778s) [noise=true , address=79.127.205.234:4619 , piece_id=3BPZ3VHTHETHWDC4XIE42QFAUKAS47T2MLYRZ3XGJ5JLTAYFXLAA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.762019154 (59h51m3.484582184s) [noise=true , address=79.127.205.246:4272 , piece_id=5B7LE4Z25OUGZGH7WQBF5KV2G2UP3QKAB5TTUUKNY2FEAQ62KTOQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.939190366 (59h51m3.661753406s) [noise=true , address=79.127.205.245:4221 , piece_id=QW4VM7F7MDOIDFQT32R5REAPLI7ZZTS4HZNFG3BEGX4BTHN6XOOQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.217973492 (59h51m3.940536532s) [noise=true , address=77.90.31.17:4796 , piece_id=VJPFUZI7545BTAW5OU2K6HAK6D2CJKJW6AMOA7LNPY5SMVVKBN2A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.339267404 (59h51m4.061830444s) [noise=true , address=156.146.43.225:4453 , piece_id=7FM3YH327FRUDPLPJM35UKUJMFQ3UPP56OSXS7MNW6B3GKBRXMWA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.462463476 (59h51m4.185026516s) [noise=true , address=79.127.219.38:4017 , piece_id=J66T5WVTOZV7MDOEE7ICJNQNPCP3EXRZWENUPQWRNR5S7RE5QK5Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:57.286145912 (59h51m7.008708952s) [noise=true , address=79.127.219.47:4048 , piece_id=ZNYUAXIUKUQRWQT536X2PEKHAORKGUNUWTSDGMP3HPAOS2LGYBLQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:59.466725590 (59h51m9.18928863s) [noise=true , address=79.127.220.101:4048 , piece_id=EY4DOTTE72JJMXUWA7MVQ2Z6ENRVPLSMVGCXRON7FMI5FDQV5TBA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:27:51.196008723 (59h52m0.918571753s) [noise=true , address=104.36.82.52:4097 , piece_id=J5VOVTTNSU2XPBTCEAPAAJOFTPM364R7CCHRVC52PAJGSKEJNH3Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: write tcp 188.166.225.170:53868->104.36.82.52:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196055173 (59h52m0.918618213s) [noise=true , address=104.36.82.22:4510 , piece_id=CVVXR3BL3V2QGZC2M3CKDMDJWWNYHQ5GZ4GI2EMYTQQXZMGGEPDA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:104.36.82.22:4510): protocol: write tcp 188.166.225.170:34290->104.36.82.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196089328 (59h52m0.918652368s) [noise=true , address=104.36.82.47:4523 , piece_id=UDBVQ2UII4ZJGW6HPMXG4UMPXW46KVN7YHFIQ5UUXGPUNQOK6O7Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 188.166.225.170:43514->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196258872 (59h52m0.918821912s) [noise=true , address=104.36.82.45:4924 , piece_id=LFONCFVCAKPEQ7PWAU5PXJ4C7FRFENACAV352WEMWRYYDETBUACA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:104.36.82.45:4924): protocol: write tcp 188.166.225.170:53286->104.36.82.45:4924: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196295432 (59h52m0.918858472s) [noise=true , address=104.36.82.53:4127 , piece_id=Y7AOQ6E64QQLOVIO4HQRWAJRMOFRFPRTYPTO4O3PBYNC2ZGNHZJQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: write tcp 188.166.225.170:46992->104.36.82.53:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196352300 (59h52m0.91891534s) [noise=true , address=104.36.82.54:4157 , piece_id=HJM5A37YBHG4O2N3W6KWBNFZWHDHOXFZ7YI3FLUXGWNV3ALX723Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196377118 (59h52m0.918940158s) [noise=true , address=104.36.82.46:4490 , piece_id=F5SINJ2OCMIH7SMEG5IOKPLGNQDGOJ3JWVBEXQ5ZV4N2FILG5R7A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 188.166.225.170:59674->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196431993 (59h52m0.918995033s) [noise=true , address=104.36.82.56:4237 , piece_id=TZOUNQCHWEOCGPJUIT3NAX7HLYSMVLHXHQIH3T37FTUQF25YLYIA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 188.166.225.170:45760->104.36.82.56:4237: use of closed network connection; conn closed: op Jul 26 23:27:51.196464665 (59h52m0.919027705s) [noise=true , address=104.36.82.21:4503 , piece_id=JROIOE56GPG2AJT7HWSZUMMZFWJ7NFB6LPB3Z3JJZBO5CC2AC3IQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:51.196483441 (59h52m0.919046481s) [seg_pos={0 0}, upload=57424]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:27:51.425876329 (59h52m1.148439369s) [upload=57424]: Done uploading segments... Jul 26 23:27:51.425900656 (59h52m1.148463696s) [upload=57424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230972543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..................................XXX.X..X.....XXXXX...XXX.X] duration: 30.001517868s FAIL (2025-07-26 23:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001736485s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.878870718s FAIL (2025-07-26 23:27:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.878901335s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.246061562s FAIL (2025-07-26 23:27:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.246082703s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [...X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....] duration: 58.700563905s FAIL (2025-07-26 23:27:14 UTC) orbiter warning: context deadline exceeded Duration: 58.700577069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235588785s FAIL (2025-07-26 23:27:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== pP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:48030->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254459354 (59h50m37.977022394s) [seg_pos={0 0}, upload=57410]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:26:28.483531338 (59h50m38.206094378s) [upload=57410]: Done uploading segments... Jul 26 23:26:28.483565944 (59h50m38.206128984s) [upload=57410]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:26:38.771605288 (59h50m48.494168318s) [upload=57420]: Starting upload Jul 26 23:26:38.771663028 (59h50m48.494226068s) [upload=57420]: Done starting upload Jul 26 23:26:38.771740014 (59h50m48.494303055s) [upload=57420]: Uploading segments... Jul 26 23:26:38.771873771 (59h50m48.494436811s) [upload=57420]: Got next segment. Inline: false Jul 26 23:26:38.771913908 (59h50m48.494476938s) [upload=57420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:26:38.771918927 (59h50m48.494481967s) [upload=57420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:26:39.006837118 (59h50m48.729400158s) [seg_pos={0 0}, upload=57420]: Begin upload segment... Jul 26 23:26:39.006891361 (59h50m48.729454401s) [seg_pos={0 0}, upload=57420]: Done begin upload segment. Jul 26 23:26:39.006912592 (59h50m48.729475622s) [noise=true , address=104.36.82.9:4007 , piece_id=P7TIRD2A5HRONEHGIYKI5ULPPOIUUYVV4I5IFPQUBLJWI75NUGSA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.006963258 (59h50m48.729526308s) [noise=true , address=45.91.116.11:4118 , piece_id=2T74KVVZVO3Q5AHCZJ67US6YZZDCS4NZTZD4K3OZMHIUYIX2NTQA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007285645 (59h50m48.729848736s) [noise=true , address=45.91.116.10:4081 , piece_id=Y5PEWGD7B2SIWPRRYXXMVE6MV4EQGP3AIVS3RFZ32XU3UR3NEGUA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007630836 (59h50m48.730193916s) [noise=true , address=45.91.116.6:4975 , piece_id=D5BQS7TOYDLZFHWDG3SDL6CN2QGTBMAYHFGTI4JM4YTDOXTDQVBQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007647027 (59h50m48.730210057s) [noise=true , address=45.91.116.14:4190 , piece_id=XD3MUL3RPM6POZL2JYASE3PZ6R4COMAONW6NLMQXRZGMUL3B2M7A, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007980264 (59h50m48.730543304s) [noise=true , address=104.36.82.11:4114 , piece_id=KCFR4KAQTOWXIRPKUJTBOGM4U6QI6L5G5KJGV463DSU7ZHUA2YXQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008315066 (59h50m48.730878096s) [noise=true , address=77.90.31.15:4698 , piece_id=ZAKGOZWCXTURYSWUNL7BNINS5FX5OGIBQGT6HFCFEVPB5NGQODLQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008757863 (59h50m48.731320903s) [noise=true , address=45.91.116.4:4940 , piece_id=QTVD733LQWLVOI4NU3LCSFTKXJ7QPA66TTNJJPEFTKUJSIORR3HQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008975409 (59h50m48.731538449s) [noise=true , address=45.91.116.13:4155 , piece_id=XMGAUAQUV433OEFZEUJE4TTTB3XLJKMWFWR3DQ7L2S2IEKAQRYKQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009195079 (59h50m48.731758119s) [noise=true , address=45.91.116.7:4010 , piece_id=EGLJ4I3WHMDKSAPN5OHMORWLNOXUNF7M3ZF5HQMPWTN7ET4JAZUQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009398478 (59h50m48.731961518s) [noise=true , address=104.36.82.5:4839 , piece_id=2QH4ZYQAAO76IJ4NAKR6KJCDAC4X2FIA4XUIVWGZYFYMEBXKB74A, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009607819 (59h50m48.732170849s) [noise=true , address=45.91.116.12:4227 , piece_id=OY6TMBDTHHOSER5MEVREOPZK6ZO4HU5SMVSNDR6Q2GO66IPTMCKQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009930716 (59h50m48.732493756s) [noise=true , address=104.36.82.4:4803 , piece_id=UGKRKHX3A6PBLOY65L4GQF2BKOQO3VIKCJRQHINIEYN5LOG3NU7Q, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010137292 (59h50m48.732700332s) [noise=true , address=104.36.82.3:4767 , piece_id=U22E5JPNZ7QT57KRX22KXYXAWX2QFMRTNWE6KR4EOUYZMMPIASUQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010322747 (59h50m48.732885777s) [noise=true , address=45.91.116.8:4047 , piece_id=HI433WOL4GHJMYSGA5CCFFSXCQIXTDM27S2X5DQUGR3VU77XHIKA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010513903 (59h50m48.733076933s) [noise=true , address=104.36.82.8:4969 , piece_id=QXI4TD4DKECLPMTCB3AVKX2ZQT4UBX26LZEIX5YS7PHFMFU65WWQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:40.734893128 (59h50m50.457456168s) [upload=57420]: Next returned nil segment Jul 26 23:26:40.734910742 (59h50m50.457473782s) [upload=57420]: Waiting for error group managing segments... Jul 26 23:26:41.011293566 (59h50m50.733856606s) [noise=true , address=45.91.116.7:4010 , piece_id=EGLJ4I3WHMDKSAPN5OHMORWLNOXUNF7M3ZF5HQMPWTN7ET4JAZUQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.042708983 (59h50m50.765272023s) [noise=true , address=45.91.116.12:4227 , piece_id=OY6TMBDTHHOSER5MEVREOPZK6ZO4HU5SMVSNDR6Q2GO66IPTMCKQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.101344720 (59h50m50.823907761s) [noise=true , address=45.91.116.13:4155 , piece_id=XMGAUAQUV433OEFZEUJE4TTTB3XLJKMWFWR3DQ7L2S2IEKAQRYKQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.110380439 (59h50m50.832943499s) [noise=true , address=45.91.116.4:4940 , piece_id=QTVD733LQWLVOI4NU3LCSFTKXJ7QPA66TTNJJPEFTKUJSIORR3HQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.117188665 (59h50m50.839751706s) [noise=true , address=45.91.116.8:4047 , piece_id=HI433WOL4GHJMYSGA5CCFFSXCQIXTDM27S2X5DQUGR3VU77XHIKA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.159493197 (59h50m50.882056237s) [noise=true , address=45.91.116.11:4118 , piece_id=2T74KVVZVO3Q5AHCZJ67US6YZZDCS4NZTZD4K3OZMHIUYIX2NTQA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.191547065 (59h50m50.914110105s) [noise=true , address=45.91.116.10:4081 , piece_id=Y5PEWGD7B2SIWPRRYXXMVE6MV4EQGP3AIVS3RFZ32XU3UR3NEGUA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.268746192 (59h50m50.991309272s) [noise=true , address=45.91.116.6:4975 , piece_id=D5BQS7TOYDLZFHWDG3SDL6CN2QGTBMAYHFGTI4JM4YTDOXTDQVBQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.587100358 (59h50m51.309663399s) [noise=true , address=45.91.116.14:4190 , piece_id=XD3MUL3RPM6POZL2JYASE3PZ6R4COMAONW6NLMQXRZGMUL3B2M7A, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:27:08.124456439 (59h51m17.847019469s) [noise=true , address=77.90.31.15:4698 , piece_id=ZAKGOZWCXTURYSWUNL7BNINS5FX5OGIBQGT6HFCFEVPB5NGQODLQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:27:08.124500803 (59h51m17.847063843s) [noise=true , address=104.36.82.4:4803 , piece_id=UGKRKHX3A6PBLOY65L4GQF2BKOQO3VIKCJRQHINIEYN5LOG3NU7Q, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 188.166.225.170:40276->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:08.124593471 (59h51m17.847156511s) [noise=true , address=104.36.82.9:4007 , piece_id=P7TIRD2A5HRONEHGIYKI5ULPPOIUUYVV4I5IFPQUBLJWI75NUGSA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124637275 (59h51m17.847200305s) [noise=true , address=104.36.82.8:4969 , piece_id=QXI4TD4DKECLPMTCB3AVKX2ZQT4UBX26LZEIX5YS7PHFMFU65WWQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124652394 (59h51m17.847215434s) [noise=true , address=104.36.82.5:4839 , piece_id=2QH4ZYQAAO76IJ4NAKR6KJCDAC4X2FIA4XUIVWGZYFYMEBXKB74A, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124700917 (59h51m17.847263957s) [noise=true , address=104.36.82.3:4767 , piece_id=U22E5JPNZ7QT57KRX22KXYXAWX2QFMRTNWE6KR4EOUYZMMPIASUQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124841726 (59h51m17.847404766s) [noise=true , address=104.36.82.11:4114 , piece_id=KCFR4KAQTOWXIRPKUJTBOGM4U6QI6L5G5KJGV463DSU7ZHUA2YXQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; context deadline exceeded Jul 26 23:27:08.124867395 (59h51m17.847430436s) [seg_pos={0 0}, upload=57420]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:27:08.358243524 (59h51m18.080806564s) [upload=57420]: Done uploading segments... Jul 26 23:27:08.358267780 (59h51m18.08083082s) [upload=57420]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235614014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.836120603s FAIL (2025-07-26 23:26:56 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.836147334s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.253141912s FAIL (2025-07-26 23:26:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.253163302s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230726148s FAIL (2025-07-26 23:26:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== piece_id=FQMQIUHWZLKSOENOPOIN7ZVRCPJHB3SHPSNHMYDUKTXCTESH6R3Q, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.821553899 (59h50m1.544116939s) [noise=true , address=104.36.82.47:4526 , piece_id=IIGEOKA4DAH4IRVTOGTJL7D7NITJ3QJEEY7UDHMMRVN4MOU366NA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.821899880 (59h50m1.54446292s) [noise=true , address=79.127.219.51:4212 , piece_id=7MIJBNKQIVZ6T5EXJTO3OZYH5PHIZVKLVMO57Y6DHPENDMPKM7ZQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822146412 (59h50m1.544709442s) [noise=true , address=77.90.31.19:4844 , piece_id=G5SLJFSN7JSJ2HRHGSAFVOJ7YZXZLP2BVKPU4U454SCTU4QSNW5A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822380250 (59h50m1.54494329s) [noise=true , address=77.90.31.16:4764 , piece_id=4USHBAIOC73YCCFP4YEF3Q4GSME3W4IKXNFIJMWD2AKGHKVZM4TQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822668661 (59h50m1.545231701s) [noise=true , address=77.90.31.21:4896 , piece_id=PLNV77UGJCLZIRPDAHD4CJGHREQCDAM4KCLUVAV3JVF4C4YL6VKQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822916886 (59h50m1.545479926s) [noise=true , address=156.146.43.225:4448 , piece_id=ERCFAATUEH2XE7ALBSGTOEIPFH25TM53EJHEWYLV4BEHUTKDMV4A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.823099245 (59h50m1.545662275s) [noise=true , address=64.49.13.138:4483 , piece_id=2IQ4426FJSTRFFLZDFISRORQYNJABGIPNCCSOHVB2INYSV4UAZRQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.826943901 (59h50m1.549506932s) [noise=true , address=104.36.82.53:4145 , piece_id=PBSW3KGSECAZU5TL7MRDU5CLXHA4RNG3VG3UX4FSQ2VTXF5CVENA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.827449258 (59h50m1.550012298s) [noise=true , address=64.49.13.138:4482 , piece_id=WBXKKSURYBGJ6PAUYI6YMZJKTAWEFE4V34CXDPZLFTLD3BNMDSSQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.828169065 (59h50m1.550732105s) [noise=true , address=79.127.205.246:4281 , piece_id=MA6GFDKRJJTUO35ZVN6ZSFAFCMYKMEPRGSIT6WLABNYW324AO4MQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.828891117 (59h50m1.551454157s) [noise=true , address=79.127.205.234:4604 , piece_id=EUGQO2NAE5S47TAA6Y77H5NUDQDIX54GNHR7RU6TFY4ASA47GYQA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.829584223 (59h50m1.552147263s) [noise=true , address=104.36.82.54:4182 , piece_id=QL6CL4GCFPHADND446NNNR2ZE74D427OGFOWZTHTTUPCQMINIC6Q, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.830273041 (59h50m1.552836081s) [noise=true , address=104.36.82.52:4109 , piece_id=YWDKYIVYVYERBSADC7THF27SDSZVQRZE67IUJ4UUKTW7TWBQPJRA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.830674710 (59h50m1.55323775s) [noise=true , address=104.36.82.46:4481 , piece_id=CPXE7BBSEZH7PLIEVSWCUVUOJS2PQVKMTUMMHSQD5SCQICAGDG3A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.831361854 (59h50m1.553924894s) [noise=true , address=79.127.205.245:4239 , piece_id=QA3GG5AXGIYCVKVXDGW5WJXPNWGGTFEPP3GH6FMIC3LLWC2SBUWA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.831652079 (59h50m1.554215109s) [noise=true , address=77.90.31.15:4722 , piece_id=Y4G5PT5SWPMLMSAFG2DQ2RLNXZ5RHFDZKP5WRA666MHPXAPURFJA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:52.842054214 (59h50m2.564617244s) [noise=true , address=156.146.43.225:4448 , piece_id=ERCFAATUEH2XE7ALBSGTOEIPFH25TM53EJHEWYLV4BEHUTKDMV4A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:25:52.928749428 (59h50m2.651312468s) [upload=57408]: Next returned nil segment Jul 26 23:25:52.928760620 (59h50m2.65132366s) [upload=57408]: Waiting for error group managing segments... Jul 26 23:25:53.207960262 (59h50m2.930523302s) [noise=true , address=79.127.205.245:4239 , piece_id=QA3GG5AXGIYCVKVXDGW5WJXPNWGGTFEPP3GH6FMIC3LLWC2SBUWA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.217235309 (59h50m2.939798349s) [noise=true , address=64.49.13.138:4482 , piece_id=WBXKKSURYBGJ6PAUYI6YMZJKTAWEFE4V34CXDPZLFTLD3BNMDSSQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:25:53.229690869 (59h50m2.952253909s) [noise=true , address=79.127.205.246:4281 , piece_id=MA6GFDKRJJTUO35ZVN6ZSFAFCMYKMEPRGSIT6WLABNYW324AO4MQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.261788051 (59h50m2.984351091s) [noise=true , address=79.127.205.234:4604 , piece_id=EUGQO2NAE5S47TAA6Y77H5NUDQDIX54GNHR7RU6TFY4ASA47GYQA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.729234437 (59h50m3.451797477s) [noise=true , address=45.91.116.9:4277 , piece_id=QHZHUJ5VTG3QTK4UYJNJIQRCKBCTPTEWIMU6W4U4YFI3AGGIE4EA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.796795362 (59h50m3.519358402s) [noise=true , address=79.127.219.38:4003 , piece_id=FQMQIUHWZLKSOENOPOIN7ZVRCPJHB3SHPSNHMYDUKTXCTESH6R3Q, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.936662861 (59h50m3.659225901s) [noise=true , address=64.49.13.138:4483 , piece_id=2IQ4426FJSTRFFLZDFISRORQYNJABGIPNCCSOHVB2INYSV4UAZRQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:56.306140127 (59h50m6.028703157s) [noise=true , address=79.127.219.51:4212 , piece_id=7MIJBNKQIVZ6T5EXJTO3OZYH5PHIZVKLVMO57Y6DHPENDMPKM7ZQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:26:23.113950228 (59h50m32.836513258s) [noise=true , address=77.90.31.19:4844 , piece_id=G5SLJFSN7JSJ2HRHGSAFVOJ7YZXZLP2BVKPU4U454SCTU4QSNW5A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:26:50.711037243 (59h51m0.433600283s) [noise=true , address=77.90.31.15:4722 , piece_id=Y4G5PT5SWPMLMSAFG2DQ2RLNXZ5RHFDZKP5WRA666MHPXAPURFJA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:26:50.711246825 (59h51m0.433809875s) [noise=true , address=104.36.82.47:4526 , piece_id=IIGEOKA4DAH4IRVTOGTJL7D7NITJ3QJEEY7UDHMMRVN4MOU366NA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711307010 (59h51m0.43387006s) [noise=true , address=77.90.31.16:4764 , piece_id=4USHBAIOC73YCCFP4YEF3Q4GSME3W4IKXNFIJMWD2AKGHKVZM4TQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711378777 (59h51m0.433941827s) [noise=true , address=77.90.31.21:4896 , piece_id=PLNV77UGJCLZIRPDAHD4CJGHREQCDAM4KCLUVAV3JVF4C4YL6VKQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711442028 (59h51m0.434005088s) [noise=true , address=104.36.82.46:4481 , piece_id=CPXE7BBSEZH7PLIEVSWCUVUOJS2PQVKMTUMMHSQD5SCQICAGDG3A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:39674->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711496893 (59h51m0.434059933s) [noise=true , address=104.36.82.54:4182 , piece_id=QL6CL4GCFPHADND446NNNR2ZE74D427OGFOWZTHTTUPCQMINIC6Q, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12HbkH31sfDUnZLXAbmzQphcFYmzB8MQo98Q6Ufbo59et42zxMy, address:104.36.82.54:4182): protocol: rpc: dial tcp 104.36.82.54:4182: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711539404 (59h51m0.434102444s) [noise=true , address=104.36.82.53:4145 , piece_id=PBSW3KGSECAZU5TL7MRDU5CLXHA4RNG3VG3UX4FSQ2VTXF5CVENA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:44082->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711619016 (59h51m0.434182046s) [noise=true , address=104.36.82.52:4109 , piece_id=YWDKYIVYVYERBSADC7THF27SDSZVQRZE67IUJ4UUKTW7TWBQPJRA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: rpc: dial tcp 104.36.82.52:4109: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711637302 (59h51m0.434200342s) [seg_pos={0 0}, upload=57408]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:26:50.939918903 (59h51m0.662481944s) [upload=57408]: Done uploading segments... Jul 26 23:26:50.939941897 (59h51m0.662504937s) [upload=57408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.230738451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.818011826s FAIL (2025-07-26 23:26:37 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.818041723s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.231831158s FAIL (2025-07-26 23:26:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 19345s) [upload=57410]: Uploading segments... Jul 26 23:25:58.901792103 (59h50m8.624355143s) [upload=57410]: Got next segment. Inline: false Jul 26 23:25:58.901832470 (59h50m8.62439551s) [upload=57410]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:25:58.901837189 (59h50m8.624400229s) [upload=57410]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:25:59.143741715 (59h50m8.866304755s) [seg_pos={0 0}, upload=57410]: Begin upload segment... Jul 26 23:25:59.143790800 (59h50m8.86635383s) [seg_pos={0 0}, upload=57410]: Done begin upload segment. Jul 26 23:25:59.143817320 (59h50m8.86638036s) [noise=true , address=45.91.116.13:4155 , piece_id=5J3LYFMYXO3GXS6Z35F3A4YBFAHD67TP75G3XHDFGS5K5PWPB2QA, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.143835766 (59h50m8.866398806s) [noise=true , address=104.36.82.14:4223 , piece_id=YTND6L3NYQYQ6JMKVJWJZEDICYZD67JGJ546SMEEXNNKJFNSOBAA, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.144177229 (59h50m8.866740269s) [noise=true , address=104.36.82.6:4876 , piece_id=UN5OC5P2CGVNC3MMEG3KOIFAXJHYXPLP56VM2QO2CSZV77Z77ASQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.144276960 (59h50m8.86684s) [noise=true , address=104.36.82.11:4113 , piece_id=M32KHVR3RQUBPWYW7URLK2ERR3VHH6G7ZDM7BMG7HKXJOPR2DGJA, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.145025452 (59h50m8.867588492s) [noise=true , address=104.36.82.13:4187 , piece_id=TE2BERM42WGWKJRTJDKPQSW4HINOW3CONCNONTA3BS53K7EQ4PJQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.145419656 (59h50m8.867982696s) [noise=true , address=45.91.116.14:4189 , piece_id=7HBJ5OV42WH4HS5JYMY5ECXTIHA3YR5Z66N56YZVAGCFYQB3CIAA, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.145812208 (59h50m8.868375238s) [noise=true , address=77.90.31.19:4841 , piece_id=7GBQ4DKAUM6BC6KIRETAAKJ5ENYMNCGX7HSA3SZMGFBWPQDTA26A, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.146616396 (59h50m8.869179436s) [noise=true , address=104.36.82.15:4257 , piece_id=JXHL2MWE4HKM5SQMOP2FOV3WRXUXSA7MFPTFAFPIURA6LUA22QCQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.147162561 (59h50m8.869725601s) [noise=true , address=77.90.31.13:4628 , piece_id=CNRWXRIMKMRIGCYXM4S33UHSP6RDCP77BVREVJYJ7H4DVOPPTMMQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.147570150 (59h50m8.87013318s) [noise=true , address=104.36.82.13:4186 , piece_id=EMUSN7D5ZMMR4ZONG7RTVCPRGQDIIVXNYOLY5KXMFWZAFUDR2RHQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.148383757 (59h50m8.870946797s) [noise=true , address=45.91.116.11:4119 , piece_id=OQYBSFO73B54LWJQ42DH5V2FBAOHOHZN5OUQ6ZBSNXZT6JYG3IHQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.148950472 (59h50m8.871513512s) [noise=true , address=104.36.82.7:4911 , piece_id=KNT72KCOWYUPUKQFOSLA7UVIBVXVVTUCFVADLZVX263TT6KR27WQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.149589785 (59h50m8.872152825s) [noise=true , address=104.36.82.7:4909 , piece_id=SS3UDQL7OI2YV5UIAGINIVVLUZWDL2LBY2WKDOZ2PGM2PFLLC5FA, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.150120470 (59h50m8.87268351s) [noise=true , address=77.90.31.14:4663 , piece_id=P3VCAGDAG2R2T6Q62SKPHENLF5L33BLPR3NRDKFLNRDJMBA5J47A, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.151032334 (59h50m8.873595374s) [noise=true , address=104.36.82.15:4258 , piece_id=F2MHYB7S2YWGTLJ35DQKLZ4M3RQTKUVQFKE6J2TGQUTJXZIKKNDQ, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:25:59.151275580 (59h50m8.87383861s) [noise=true , address=45.91.116.10:4081 , piece_id=VC7QLJ6RMBPG7DFS3Z444UYZIVHLA2CYYZQ4OTOGIXQ7Z465PZ5A, seg_pos={0 0}, upload=57410]: Uploading piece... Jul 26 23:26:00.890230476 (59h50m10.612793516s) [upload=57410]: Next returned nil segment Jul 26 23:26:00.890246747 (59h50m10.612809777s) [upload=57410]: Waiting for error group managing segments... Jul 26 23:26:01.276988600 (59h50m10.99955164s) [noise=true , address=77.90.31.14:4663 , piece_id=P3VCAGDAG2R2T6Q62SKPHENLF5L33BLPR3NRDKFLNRDJMBA5J47A, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:01.294914649 (59h50m11.017477689s) [noise=true , address=45.91.116.14:4189 , piece_id=7HBJ5OV42WH4HS5JYMY5ECXTIHA3YR5Z66N56YZVAGCFYQB3CIAA, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:01.309966207 (59h50m11.032529247s) [noise=true , address=45.91.116.13:4155 , piece_id=5J3LYFMYXO3GXS6Z35F3A4YBFAHD67TP75G3XHDFGS5K5PWPB2QA, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:01.344942637 (59h50m11.067505677s) [noise=true , address=45.91.116.11:4119 , piece_id=OQYBSFO73B54LWJQ42DH5V2FBAOHOHZN5OUQ6ZBSNXZT6JYG3IHQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:01.429774416 (59h50m11.152337456s) [noise=true , address=45.91.116.10:4081 , piece_id=VC7QLJ6RMBPG7DFS3Z444UYZIVHLA2CYYZQ4OTOGIXQ7Z465PZ5A, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:02.579643804 (59h50m12.302206854s) [noise=true , address=77.90.31.13:4628 , piece_id=CNRWXRIMKMRIGCYXM4S33UHSP6RDCP77BVREVJYJ7H4DVOPPTMMQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:03.401122741 (59h50m13.123685781s) [noise=true , address=77.90.31.19:4841 , piece_id=7GBQ4DKAUM6BC6KIRETAAKJ5ENYMNCGX7HSA3SZMGFBWPQDTA26A, seg_pos={0 0}, upload=57410]: Done uploading piece. err: <nil> Jul 26 23:26:28.253824479 (59h50m37.976387519s) [noise=true , address=104.36.82.15:4257 , piece_id=JXHL2MWE4HKM5SQMOP2FOV3WRXUXSA7MFPTFAFPIURA6LUA22QCQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.253929601 (59h50m37.976492641s) [noise=true , address=104.36.82.7:4909 , piece_id=SS3UDQL7OI2YV5UIAGINIVVLUZWDL2LBY2WKDOZ2PGM2PFLLC5FA, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.253968966 (59h50m37.976532006s) [noise=true , address=104.36.82.15:4258 , piece_id=F2MHYB7S2YWGTLJ35DQKLZ4M3RQTKUVQFKE6J2TGQUTJXZIKKNDQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254132959 (59h50m37.976695999s) [noise=true , address=104.36.82.14:4223 , piece_id=YTND6L3NYQYQ6JMKVJWJZEDICYZD67JGJ546SMEEXNNKJFNSOBAA, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254232600 (59h50m37.97679564s) [noise=true , address=104.36.82.7:4911 , piece_id=KNT72KCOWYUPUKQFOSLA7UVIBVXVVTUCFVADLZVX263TT6KR27WQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254300820 (59h50m37.976863851s) [noise=true , address=104.36.82.13:4187 , piece_id=TE2BERM42WGWKJRTJDKPQSW4HINOW3CONCNONTA3BS53K7EQ4PJQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254339405 (59h50m37.976902445s) [noise=true , address=104.36.82.13:4186 , piece_id=EMUSN7D5ZMMR4ZONG7RTVCPRGQDIIVXNYOLY5KXMFWZAFUDR2RHQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:55770->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254389321 (59h50m37.976952361s) [noise=true , address=104.36.82.11:4113 , piece_id=M32KHVR3RQUBPWYW7URLK2ERR3VHH6G7ZDM7BMG7HKXJOPR2DGJA, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254435689 (59h50m37.976998719s) [noise=true , address=104.36.82.6:4876 , piece_id=UN5OC5P2CGVNC3MMEG3KOIFAXJHYXPLP56VM2QO2CSZV77Z77ASQ, seg_pos={0 0}, upload=57410]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:48030->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254459354 (59h50m37.977022394s) [seg_pos={0 0}, upload=57410]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:26:28.483531338 (59h50m38.206094378s) [upload=57410]: Done uploading segments... Jul 26 23:26:28.483565944 (59h50m38.206128984s) [upload=57410]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.231879571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [......X.X...........................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232005565s FAIL (2025-07-26 23:26:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.23203353s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [....X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX....] duration: 45.52361522s FAIL (2025-07-26 23:26:15 UTC) orbiter warning: context deadline exceeded Duration: 45.523655898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................................XXX.X..X.....XXXXX...XXX.] duration: 8.041689751s FAIL (2025-07-26 23:26:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.04171041s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229890573s FAIL (2025-07-26 23:25:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ad=57392]: Begin upload segment... Jul 26 23:24:51.338188250 (59h49m1.06075129s) [seg_pos={0 0}, upload=57392]: Done begin upload segment. Jul 26 23:24:51.338210472 (59h49m1.060773513s) [noise=true , address=77.90.31.18:4829 , piece_id=QPDU74R2MOR4TYKUWI4PSQFEU2PVBM6W2U54P6AFTEFMW4D6KH4A, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.338270167 (59h49m1.060833227s) [noise=true , address=104.36.82.56:4262 , piece_id=CDWVEZGLM64JVVIRZ75FGGRT47PXLS2LKFDGX7AMOMA6FX3U37OA, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.338578967 (59h49m1.061142007s) [noise=true , address=77.90.31.15:4703 , piece_id=SQPJRWCVXKTQY2WMHAXSNQVROLITMESHKVXSPPMMH5KKFGA7UG6Q, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.338839296 (59h49m1.061402336s) [noise=true , address=79.127.219.51:4212 , piece_id=LLSZAALHQ7D76BNN2ACMRSOJ6ETXB2W2TOFJZELISWZ7IKTC654Q, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339037535 (59h49m1.061600585s) [noise=true , address=104.36.82.50:4022 , piece_id=CBNOSNGDEKPWZKI2OCTFHDLXU6GV44YIXYF2GRPAX4TJXBSG6CIA, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339106677 (59h49m1.061669717s) [noise=true , address=79.127.205.245:4218 , piece_id=Q2V4DTYKKX45KJCEXJ3WNZSPQRNMYXYHPFNYZYPXIRL7OXNCMD5A, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339363278 (59h49m1.061926318s) [noise=true , address=104.36.82.47:4526 , piece_id=QPI3PV24TM265SATOSHCKAQXED4M5VVCFKPZY4226HSF7VARLPNQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339524517 (59h49m1.062087567s) [noise=true , address=64.49.13.138:4482 , piece_id=XUHOJ3QYHHDBB53FRZXNVRTHZ3RKWDUX7ONY3SV5F4ZDK52N7TQQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339600391 (59h49m1.062163431s) [noise=true , address=79.127.205.235:4682 , piece_id=BAJEKS3M7SIBPDVMZAYK6CCXUXIA3MVSQQQIEMM34OE43ZAAS6MQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.339829671 (59h49m1.062392711s) [noise=true , address=156.146.43.225:4448 , piece_id=MJM5CWUJETKLNMWCK54DKIJKIDLWWNVJX2BQCIHPGCT7JNFANYQQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.340035744 (59h49m1.062598784s) [noise=true , address=104.36.82.46:4498 , piece_id=OY4IYBENVLUNUK5XCZNLBQJKVJN246INBPNNDEGWXAORUODB7ABA, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.340270885 (59h49m1.062833925s) [noise=true , address=79.127.201.215:4345 , piece_id=KLKSQW2U2FSYP3RAU2AGJZIGDCXX7QAKIRAPENYQMVHPXXHNZJUA, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.340475075 (59h49m1.063038115s) [noise=true , address=104.36.82.54:4167 , piece_id=7YUUPH3QJIVLDQDAPEAT6Q6ML2YF7GJ5O6H42MBWQ72D5PNWNADQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.340737207 (59h49m1.063300257s) [noise=true , address=45.91.116.5:4307 , piece_id=E5JBYHTPUSPFOIWDSSYWRJVVUDE27ZXCX45TT4IXUQX2DZU2AL6A, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.340902603 (59h49m1.063465633s) [noise=true , address=79.127.220.97:4361 , piece_id=HV7WY77XGA4I5COWI7SC5VGAORPPWZFTP7ASS2GI5HLT6ERAKWPQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.341200704 (59h49m1.063763744s) [noise=true , address=104.36.82.53:4148 , piece_id=7HBRTHK6KG6CUVQHLDXWPD23A6U3R32AA4YNHI4HGA7OFIZY2OMA, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.341447675 (59h49m1.064010716s) [noise=true , address=77.90.31.16:4736 , piece_id=UTK5T46O3WMAKGANE7ZIV3GB6XITAAX6VGD6JS3BXDPUWIDW7NVQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:51.341739825 (59h49m1.064302865s) [noise=true , address=79.127.205.234:4612 , piece_id=YI7WBTVX2EN4YLQRKRZJTCYTT6MYXPL5QGNPOAWCVMB7JYIYZBJQ, seg_pos={0 0}, upload=57392]: Uploading piece... Jul 26 23:24:52.431841619 (59h49m2.154404659s) [upload=57392]: Next returned nil segment Jul 26 23:24:52.431856106 (59h49m2.154419146s) [upload=57392]: Waiting for error group managing segments... Jul 26 23:24:52.742553774 (59h49m2.465116804s) [noise=true , address=79.127.205.245:4218 , piece_id=Q2V4DTYKKX45KJCEXJ3WNZSPQRNMYXYHPFNYZYPXIRL7OXNCMD5A, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:52.746711891 (59h49m2.469274931s) [noise=true , address=79.127.205.235:4682 , piece_id=BAJEKS3M7SIBPDVMZAYK6CCXUXIA3MVSQQQIEMM34OE43ZAAS6MQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:52.749465549 (59h49m2.472028579s) [noise=true , address=79.127.205.234:4612 , piece_id=YI7WBTVX2EN4YLQRKRZJTCYTT6MYXPL5QGNPOAWCVMB7JYIYZBJQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:53.276609876 (59h49m2.999172916s) [noise=true , address=77.90.31.15:4703 , piece_id=SQPJRWCVXKTQY2WMHAXSNQVROLITMESHKVXSPPMMH5KKFGA7UG6Q, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:53.407717103 (59h49m3.130280143s) [noise=true , address=77.90.31.18:4829 , piece_id=QPDU74R2MOR4TYKUWI4PSQFEU2PVBM6W2U54P6AFTEFMW4D6KH4A, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:54.901703230 (59h49m4.62426627s) [noise=true , address=64.49.13.138:4482 , piece_id=XUHOJ3QYHHDBB53FRZXNVRTHZ3RKWDUX7ONY3SV5F4ZDK52N7TQQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:55.218166670 (59h49m4.94072971s) [noise=true , address=79.127.219.51:4212 , piece_id=LLSZAALHQ7D76BNN2ACMRSOJ6ETXB2W2TOFJZELISWZ7IKTC654Q, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:56.205955908 (59h49m5.928518948s) [noise=true , address=45.91.116.5:4307 , piece_id=E5JBYHTPUSPFOIWDSSYWRJVVUDE27ZXCX45TT4IXUQX2DZU2AL6A, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:24:57.645722277 (59h49m7.368285317s) [noise=true , address=156.146.43.225:4448 , piece_id=MJM5CWUJETKLNMWCK54DKIJKIDLWWNVJX2BQCIHPGCT7JNFANYQQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:25:01.583850968 (59h49m11.306414018s) [noise=true , address=79.127.220.97:4361 , piece_id=HV7WY77XGA4I5COWI7SC5VGAORPPWZFTP7ASS2GI5HLT6ERAKWPQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:25:11.757012739 (59h49m21.479575799s) [noise=true , address=79.127.201.215:4345 , piece_id=KLKSQW2U2FSYP3RAU2AGJZIGDCXX7QAKIRAPENYQMVHPXXHNZJUA, seg_pos={0 0}, upload=57392]: Done uploading piece. err: <nil> Jul 26 23:25:50.250460833 (59h49m59.973023873s) [noise=true , address=104.36.82.53:4148 , piece_id=7HBRTHK6KG6CUVQHLDXWPD23A6U3R32AA4YNHI4HGA7OFIZY2OMA, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250507062 (59h49m59.973070102s) [noise=true , address=104.36.82.46:4498 , piece_id=OY4IYBENVLUNUK5XCZNLBQJKVJN246INBPNNDEGWXAORUODB7ABA, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: rpc: dial tcp 104.36.82.46:4498: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250535837 (59h49m59.973099067s) [noise=true , address=104.36.82.54:4167 , piece_id=7YUUPH3QJIVLDQDAPEAT6Q6ML2YF7GJ5O6H42MBWQ72D5PNWNADQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: rpc: dial tcp 104.36.82.54:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250632201 (59h49m59.973195241s) [noise=true , address=77.90.31.16:4736 , piece_id=UTK5T46O3WMAKGANE7ZIV3GB6XITAAX6VGD6JS3BXDPUWIDW7NVQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250642291 (59h49m59.973205321s) [noise=true , address=104.36.82.56:4262 , piece_id=CDWVEZGLM64JVVIRZ75FGGRT47PXLS2LKFDGX7AMOMA6FX3U37OA, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: write tcp 188.166.225.170:37662->104.36.82.56:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250678940 (59h49m59.97324198s) [noise=true , address=104.36.82.47:4526 , piece_id=QPI3PV24TM265SATOSHCKAQXED4M5VVCFKPZY4226HSF7VARLPNQ, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:42680->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250697747 (59h49m59.973260777s) [noise=true , address=104.36.82.50:4022 , piece_id=CBNOSNGDEKPWZKI2OCTFHDLXU6GV44YIXYF2GRPAX4TJXBSG6CIA, seg_pos={0 0}, upload=57392]: Done uploading piece. err: ecclient: upload failed (node:12ZPU1MyQ2Adj32UCvyosUGEEpKzg9riU9izZxqRuXRtEJLfse1, address:104.36.82.50:4022): protocol: write tcp 188.166.225.170:53494->104.36.82.50:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:50.250721802 (59h49m59.973284912s) [seg_pos={0 0}, upload=57392]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:25:50.478995068 (59h50m0.201558108s) [upload=57392]: Done uploading segments... Jul 26 23:25:50.479022180 (59h50m0.20158522s) [upload=57392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.229928947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................................XXX.X..X.....XXXXX...XXX] duration: 30.00083192s FAIL (2025-07-26 23:25:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000842951s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.892209761s FAIL (2025-07-26 23:25:31 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.892241362s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [....X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237368553s FAIL (2025-07-26 23:25:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 395]: Done begin upload segment. Jul 26 23:25:00.996861259 (59h49m10.719424299s) [noise=true , address=104.36.82.4:4803 , piece_id=YFCGWE5VMJZZYMLUDOAN7XQPDGZ4LOWQZMJ5BHJEMF6YDPAHKA7A, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.996869415 (59h49m10.719432455s) [noise=true , address=104.36.82.12:4151 , piece_id=GCW6V7MS33DLVV7JX6ZE5BMDSEOXJ36MGXZ4C7H6PPSZVLFYZW7Q, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997192332 (59h49m10.719755372s) [noise=true , address=104.36.82.5:4839 , piece_id=OZRZGCRFI6WULPO5BGWX65DG2S36UKOZD4A4P2G3FNKU2HVSKTXA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997269830 (59h49m10.719832871s) [noise=true , address=104.36.82.9:4005 , piece_id=NN33DUGWNGBR3QVT7KYIBQFBFPWZROUMZGX5EMMBTSEUWHBBRMTQ, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997458702 (59h49m10.720021742s) [noise=true , address=104.36.82.6:4873 , piece_id=X4VRYTASKJRSY4PULF6LO75JSQVEWB7UMAJFVNWYW6DLWCDSM7JA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997584453 (59h49m10.720147493s) [noise=true , address=45.91.116.9:4261 , piece_id=3NREONQU33FXOKD75KNNBCLYQU2CV2APK5GYDPJCZAXLUC3RAHVA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997710514 (59h49m10.720273554s) [noise=true , address=104.36.82.16:4295 , piece_id=Q6J466PJZMDSJRIXTI5J4XP3KTZ65DHCBUQCUKWLCL3FBZ2WU5ZQ, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997923111 (59h49m10.720486151s) [noise=true , address=77.90.31.14:4662 , piece_id=EN36WY7RB7PVKTXVLUXVQF4OGNQHC6DMGE4V7VCAJAQZOURRDDYA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.997981823 (59h49m10.720544853s) [noise=true , address=77.90.31.19:4842 , piece_id=K4OW7MQCYRT4PC5TUZNEZTZ6F65WVVYWHICPKWNEI23SR2MHJ5OA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998208256 (59h49m10.720771296s) [noise=true , address=104.36.82.8:4972 , piece_id=A3X7UL5SGHQNXANNDNA43OC42HGMJPTKMUPXCBUY4YRN4AJKUMAQ, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998229627 (59h49m10.720792667s) [noise=true , address=45.91.116.8:4048 , piece_id=VIDK7LZOEDD4IRKKXORNWE3W4FFMHUNSUE3P2G2MJ2HCHDZAEGJA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998408289 (59h49m10.720971329s) [noise=true , address=104.36.82.6:4876 , piece_id=W5DH7HJKRDRHIKSNGNJKVTUHWTKSQ7WXXRBBEREURHEBLTQHS3GQ, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998534290 (59h49m10.72109734s) [noise=true , address=104.36.82.14:4223 , piece_id=4YFCYOB4UARS2B46TQSDLVVNNM5IUKG6ZZV3VJNAAK72JULHSL6A, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998666192 (59h49m10.721229222s) [noise=true , address=104.36.82.13:4186 , piece_id=NCCARNFCV2LR4YKYOLI2V672F2JGVHTT5DCRTUW437LL4CJMAZ6Q, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998821490 (59h49m10.72138453s) [noise=true , address=104.36.82.10:4077 , piece_id=4BW2XHT6XW627QPRPSVRXRMFJNJ4G5O4NQ35PYJCXU773JRDENRA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:00.998870453 (59h49m10.721433493s) [noise=true , address=77.90.31.15:4699 , piece_id=EYG4T4EM2O6Q5WMCFT3NDAFRI3L3BYLT3BRHEIFEUAMGTI27ITGA, seg_pos={0 0}, upload=57395]: Uploading piece... Jul 26 23:25:02.737641177 (59h49m12.460204217s) [upload=57395]: Next returned nil segment Jul 26 23:25:02.737661967 (59h49m12.460225017s) [upload=57395]: Waiting for error group managing segments... Jul 26 23:25:03.033047919 (59h49m12.755610959s) [noise=true , address=77.90.31.14:4662 , piece_id=EN36WY7RB7PVKTXVLUXVQF4OGNQHC6DMGE4V7VCAJAQZOURRDDYA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: <nil> Jul 26 23:25:03.095193873 (59h49m12.817756913s) [noise=true , address=45.91.116.8:4048 , piece_id=VIDK7LZOEDD4IRKKXORNWE3W4FFMHUNSUE3P2G2MJ2HCHDZAEGJA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: <nil> Jul 26 23:25:03.464075354 (59h49m13.186638394s) [noise=true , address=45.91.116.9:4261 , piece_id=3NREONQU33FXOKD75KNNBCLYQU2CV2APK5GYDPJCZAXLUC3RAHVA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: <nil> Jul 26 23:25:04.161901887 (59h49m13.884464928s) [noise=true , address=77.90.31.15:4699 , piece_id=EYG4T4EM2O6Q5WMCFT3NDAFRI3L3BYLT3BRHEIFEUAMGTI27ITGA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: <nil> Jul 26 23:25:05.233725330 (59h49m14.95628836s) [noise=true , address=77.90.31.19:4842 , piece_id=K4OW7MQCYRT4PC5TUZNEZTZ6F65WVVYWHICPKWNEI23SR2MHJ5OA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: <nil> Jul 26 23:25:30.109328388 (59h49m39.831891428s) [noise=true , address=104.36.82.16:4295 , piece_id=Q6J466PJZMDSJRIXTI5J4XP3KTZ65DHCBUQCUKWLCL3FBZ2WU5ZQ, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46622->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109384205 (59h49m39.831947245s) [noise=true , address=104.36.82.8:4972 , piece_id=A3X7UL5SGHQNXANNDNA43OC42HGMJPTKMUPXCBUY4YRN4AJKUMAQ, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:55302->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109434611 (59h49m39.831997651s) [noise=true , address=104.36.82.10:4077 , piece_id=4BW2XHT6XW627QPRPSVRXRMFJNJ4G5O4NQ35PYJCXU773JRDENRA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:47540->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109512280 (59h49m39.83207532s) [noise=true , address=104.36.82.4:4803 , piece_id=YFCGWE5VMJZZYMLUDOAN7XQPDGZ4LOWQZMJ5BHJEMF6YDPAHKA7A, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109566614 (59h49m39.832129654s) [noise=true , address=104.36.82.12:4151 , piece_id=GCW6V7MS33DLVV7JX6ZE5BMDSEOXJ36MGXZ4C7H6PPSZVLFYZW7Q, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109598855 (59h49m39.832161896s) [noise=true , address=104.36.82.9:4005 , piece_id=NN33DUGWNGBR3QVT7KYIBQFBFPWZROUMZGX5EMMBTSEUWHBBRMTQ, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109632910 (59h49m39.83219594s) [noise=true , address=104.36.82.6:4876 , piece_id=W5DH7HJKRDRHIKSNGNJKVTUHWTKSQ7WXXRBBEREURHEBLTQHS3GQ, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109656175 (59h49m39.832219205s) [noise=true , address=104.36.82.13:4186 , piece_id=NCCARNFCV2LR4YKYOLI2V672F2JGVHTT5DCRTUW437LL4CJMAZ6Q, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109688387 (59h49m39.832251417s) [noise=true , address=104.36.82.6:4873 , piece_id=X4VRYTASKJRSY4PULF6LO75JSQVEWB7UMAJFVNWYW6DLWCDSM7JA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109741789 (59h49m39.832304829s) [noise=true , address=104.36.82.14:4223 , piece_id=4YFCYOB4UARS2B46TQSDLVVNNM5IUKG6ZZV3VJNAAK72JULHSL6A, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109832312 (59h49m39.832395352s) [noise=true , address=104.36.82.5:4839 , piece_id=OZRZGCRFI6WULPO5BGWX65DG2S36UKOZD4A4P2G3FNKU2HVSKTXA, seg_pos={0 0}, upload=57395]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:44676->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:30.109883279 (59h49m39.832446329s) [seg_pos={0 0}, upload=57395]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:25:30.345406946 (59h49m40.067970026s) [upload=57395]: Done uploading segments... Jul 26 23:25:30.345438076 (59h49m40.068001116s) [upload=57395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.23738227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.....X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX...] duration: 34.559129662s FAIL (2025-07-26 23:25:17 UTC) orbiter warning: context deadline exceeded Duration: 34.559149662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.......X.X...........................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237273516s FAIL (2025-07-26 23:25:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.237290949s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................................XXX.X..X.....XXXXX...XX] duration: 30.001331818s FAIL (2025-07-26 23:25:04 UTC) orbiter warning: context deadline exceeded Duration: 30.001363207s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.....X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236446994s FAIL (2025-07-26 23:24:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== :30.281122988 (59h48m40.003686018s) [upload=57383]: Uploading segments... Jul 26 23:24:30.281170218 (59h48m40.003733258s) [upload=57383]: Got next segment. Inline: false Jul 26 23:24:30.281192170 (59h48m40.00375521s) [upload=57383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:24:30.281198302 (59h48m40.003761332s) [upload=57383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:24:30.514839206 (59h48m40.237402256s) [seg_pos={0 0}, upload=57383]: Begin upload segment... Jul 26 23:24:30.514890604 (59h48m40.237453644s) [seg_pos={0 0}, upload=57383]: Done begin upload segment. Jul 26 23:24:30.514915773 (59h48m40.237478823s) [noise=true , address=45.91.116.11:4118 , piece_id=2O7AFSFXZEKWDIN4DMU7ERZ6B3ZSDJF5IEZ6CXAF7FI2YHY75FCA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.515357257 (59h48m40.237920297s) [noise=true , address=104.36.82.9:4007 , piece_id=4VXGRMWZHX6K2ABZVWQDMEYBT33F54UQHIWHUVYDKY7URXQTXBLA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.515717196 (59h48m40.238280236s) [noise=true , address=104.36.82.5:4838 , piece_id=E6K5SUFJ2ODGWKIQFFRLVU4DXFKIGYMFDVOR3FDKP5JEVQDPNV3A, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.516063638 (59h48m40.238626688s) [noise=true , address=104.36.82.4:4801 , piece_id=RJWCRL7K7AFBNZ53EE3YHMKRAGT5CVNZGSE3AHDWR74NX2NFLA4A, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.516449046 (59h48m40.239012086s) [noise=true , address=104.36.82.2:4731 , piece_id=JF2IOHQDQIEV6INKKGGICF5QBIDIXH4DNWUW2776HY3DYOULDBMQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.516779398 (59h48m40.239342438s) [noise=true , address=104.36.82.13:4185 , piece_id=HNAMSA3YTS77IBZAGPIK347USRX6XUXEPV5HUPSD2FHQAUQNNBAA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.517077758 (59h48m40.239640799s) [noise=true , address=45.91.116.10:4083 , piece_id=CN6EMTRZBJ3LMXF75GWM4C4VPGDUB4TR2X5ZR2LRJPASZK7VTNLQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.517382132 (59h48m40.239945172s) [noise=true , address=45.91.116.13:4153 , piece_id=5CV4UHAP77HZRFWIMUB6JI64CTWUM4PD4M6ANVT2MBBZZPZM6BWA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.517684961 (59h48m40.240248001s) [noise=true , address=45.91.116.7:4011 , piece_id=W7V6VZ4EIBL6D7TXTXQBDJ2DM44YJM5PHPYSFFRN63BPWYOOZITQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.518083233 (59h48m40.240646273s) [noise=true , address=104.36.82.15:4258 , piece_id=RJLCYYXVZBTE6AZPU3T6732CPKNW4SIJJ3XP7MP5DMSIYJMFR3MQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.518378929 (59h48m40.240941969s) [noise=true , address=104.36.82.6:4874 , piece_id=NAECL5J2AMJH6V4ZNJS4TACSSCTQ2GCF5SRIZW5PASM4B5GFPWEA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.518691096 (59h48m40.241254126s) [noise=true , address=77.90.31.17:4771 , piece_id=MIRKJ5N2NUWZLUMA46JRLR73AHGVLNKIYSTNQVQ2VNYOPNS3JU4A, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.519073527 (59h48m40.241636567s) [noise=true , address=77.90.31.14:4661 , piece_id=PN6MK2DDCTTLVLV433LVR56USTUMHBJ6JWTHGI73UCHGG5BO2ZWQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.519498741 (59h48m40.242061782s) [noise=true , address=104.36.82.2:4730 , piece_id=GID653XHU7NOOBIYOIUECHDDPP4YC4YWQZAH37UO7AOP3IAVERVA, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.519790078 (59h48m40.242353108s) [noise=true , address=45.91.116.14:4189 , piece_id=JBNJLBVUG4ALOFEJPQKNLOYF7ORREOV4GCL63ZIS66CXWGFD3DKQ, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:30.519992256 (59h48m40.242555296s) [noise=true , address=104.36.82.14:4221 , piece_id=ENMTE2OTNU2PICI2WZLIEYTEEPLWF4EDHHVSL5AO26EIZGSPN76A, seg_pos={0 0}, upload=57383]: Uploading piece... Jul 26 23:24:32.151391976 (59h48m41.873955016s) [upload=57383]: Next returned nil segment Jul 26 23:24:32.151407906 (59h48m41.873970946s) [upload=57383]: Waiting for error group managing segments... Jul 26 23:24:32.541035372 (59h48m42.263598412s) [noise=true , address=77.90.31.14:4661 , piece_id=PN6MK2DDCTTLVLV433LVR56USTUMHBJ6JWTHGI73UCHGG5BO2ZWQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:32.551357541 (59h48m42.273920641s) [noise=true , address=45.91.116.7:4011 , piece_id=W7V6VZ4EIBL6D7TXTXQBDJ2DM44YJM5PHPYSFFRN63BPWYOOZITQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:32.571340625 (59h48m42.293903665s) [noise=true , address=45.91.116.14:4189 , piece_id=JBNJLBVUG4ALOFEJPQKNLOYF7ORREOV4GCL63ZIS66CXWGFD3DKQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:32.680568661 (59h48m42.403131701s) [noise=true , address=45.91.116.13:4153 , piece_id=5CV4UHAP77HZRFWIMUB6JI64CTWUM4PD4M6ANVT2MBBZZPZM6BWA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:32.820618719 (59h48m42.543181759s) [noise=true , address=45.91.116.11:4118 , piece_id=2O7AFSFXZEKWDIN4DMU7ERZ6B3ZSDJF5IEZ6CXAF7FI2YHY75FCA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:32.841727156 (59h48m42.564290226s) [noise=true , address=45.91.116.10:4083 , piece_id=CN6EMTRZBJ3LMXF75GWM4C4VPGDUB4TR2X5ZR2LRJPASZK7VTNLQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:55.545416758 (59h49m5.267979798s) [noise=true , address=77.90.31.17:4771 , piece_id=MIRKJ5N2NUWZLUMA46JRLR73AHGVLNKIYSTNQVQ2VNYOPNS3JU4A, seg_pos={0 0}, upload=57383]: Done uploading piece. err: <nil> Jul 26 23:24:59.641517248 (59h49m9.364080288s) [noise=true , address=104.36.82.15:4258 , piece_id=RJLCYYXVZBTE6AZPU3T6732CPKNW4SIJJ3XP7MP5DMSIYJMFR3MQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641565921 (59h49m9.364128961s) [noise=true , address=104.36.82.6:4874 , piece_id=NAECL5J2AMJH6V4ZNJS4TACSSCTQ2GCF5SRIZW5PASM4B5GFPWEA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641596269 (59h49m9.364159309s) [noise=true , address=104.36.82.2:4730 , piece_id=GID653XHU7NOOBIYOIUECHDDPP4YC4YWQZAH37UO7AOP3IAVERVA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641653849 (59h49m9.364216889s) [noise=true , address=104.36.82.5:4838 , piece_id=E6K5SUFJ2ODGWKIQFFRLVU4DXFKIGYMFDVOR3FDKP5JEVQDPNV3A, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:54392->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641831429 (59h49m9.364394459s) [noise=true , address=104.36.82.4:4801 , piece_id=RJWCRL7K7AFBNZ53EE3YHMKRAGT5CVNZGSE3AHDWR74NX2NFLA4A, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641868941 (59h49m9.364431981s) [noise=true , address=104.36.82.14:4221 , piece_id=ENMTE2OTNU2PICI2WZLIEYTEEPLWF4EDHHVSL5AO26EIZGSPN76A, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641913255 (59h49m9.364476295s) [noise=true , address=104.36.82.13:4185 , piece_id=HNAMSA3YTS77IBZAGPIK347USRX6XUXEPV5HUPSD2FHQAUQNNBAA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641920430 (59h49m9.36448347s) [noise=true , address=104.36.82.9:4007 , piece_id=4VXGRMWZHX6K2ABZVWQDMEYBT33F54UQHIWHUVYDKY7URXQTXBLA, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641946048 (59h49m9.364509088s) [noise=true , address=104.36.82.2:4731 , piece_id=JF2IOHQDQIEV6INKKGGICF5QBIDIXH4DNWUW2776HY3DYOULDBMQ, seg_pos={0 0}, upload=57383]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:59.641971176 (59h49m9.364534216s) [seg_pos={0 0}, upload=57383]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:24:59.876815192 (59h49m9.599378232s) [upload=57383]: Done uploading segments... Jul 26 23:24:59.876838908 (59h49m9.599401948s) [upload=57383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.236460169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.848124931s FAIL (2025-07-26 23:24:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.848155549s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229084885s FAIL (2025-07-26 23:24:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== load=57374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:23:50.871991408 (59h48m0.594554438s) [seg_pos={0 0}, upload=57374]: Begin upload segment... Jul 26 23:23:50.872040142 (59h48m0.594603172s) [seg_pos={0 0}, upload=57374]: Done begin upload segment. Jul 26 23:23:50.872063316 (59h48m0.594626356s) [noise=true , address=64.49.13.140:4508 , piece_id=MIANH7HBKK7MIWLRYW4G73X546NBPEBRDR4KSQYLUFZ4RM4O2F3A, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872104364 (59h48m0.594667404s) [noise=true , address=104.36.82.55:4193 , piece_id=DGARUIVD3TQN5TR3TEIO3YJ5OUPJNSPWED77QNAGTHNTFUNGQZNA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872443213 (59h48m0.595006243s) [noise=true , address=64.49.13.140:4514 , piece_id=MRXU3VAKV5JH4GA7AVTQRTGPX2MVG6FHY6W5IRR5CLUSLBXSNSCQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872534087 (59h48m0.595097127s) [noise=true , address=79.127.219.47:4061 , piece_id=I64NW4VOJJAIQACPII6RRFA4WDX4Z6IUPSNBC3BCCYYJXTL547TA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872711746 (59h48m0.595274786s) [noise=true , address=104.36.82.47:4549 , piece_id=5VHFQCLA6OLE5YDE5MJEGADQJTXLBICGEWE3RBKU7ERU5EHLRPPA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872806919 (59h48m0.595369959s) [noise=true , address=104.36.82.56:4231 , piece_id=LDPTNPZOX7E5KI5IW7SVWMG7NOJ34IS3A52LKVWXZ5JDTVBWIEAQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.872976173 (59h48m0.595539203s) [noise=true , address=79.127.205.246:4274 , piece_id=L3DV63UNNCT2M7WSSYLZGVHKUVW4I33OCHA2PHD3EE7ZQMILFZOA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.873237213 (59h48m0.595800253s) [noise=true , address=79.127.205.235:4690 , piece_id=EKY3LQAH2KZT4XQF5UPTTHK5DA2NJMAHEYVNTH74ESCI35EWBGLA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.873475277 (59h48m0.596038307s) [noise=true , address=104.36.82.46:4507 , piece_id=XVKL6DI2P46BKN2Q6PC3WDMPVXNE2MN4SXDTWTFNLST7LAFOTV4Q, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.873702603 (59h48m0.596265643s) [noise=true , address=77.90.31.21:4887 , piece_id=66L5AAOBJ7MPQ4NGOZ75RAOFOQQRGKNY5E7NU6D3JSNA5TPVKYBQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.873941319 (59h48m0.596504359s) [noise=true , address=79.127.205.245:4223 , piece_id=EMR2ADKGWBRFFV3JJADP7RQKKJLF4C2JMX5FWA6GAAOCNHMU5SHA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.874161771 (59h48m0.596724811s) [noise=true , address=64.49.13.138:4483 , piece_id=MLLP3XDISSNKZX7W3VF7E3JPHHV2GONSXWHVMDNHREMLNLKQBZ4A, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.874376302 (59h48m0.596939342s) [noise=true , address=104.36.82.53:4142 , piece_id=BP3IKS4PSNGKLQXTXTPVIEACBBCC4QPALJ56VZMQ7CWNEVJLGVHQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.874591293 (59h48m0.597154333s) [noise=true , address=45.91.116.14:4206 , piece_id=3ZDG5RILWVTXD3OIQOFBDPV5ETL52JGLIQA5ICKXOIFJNIRJWLNA, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.874775105 (59h48m0.597338145s) [noise=true , address=79.127.205.234:4591 , piece_id=MO5C6KOCIBGYLZJG4LRMZRHVXFHXZZJAT5X4E7HB4S2UMOPOAYVQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.874988664 (59h48m0.597551704s) [noise=true , address=79.127.219.51:4210 , piece_id=3JF445FCNFCHZ2YY3NK5T4SXIAP22Z7FKE33IYWJYERYDNSRLUJQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.875187705 (59h48m0.597750745s) [noise=true , address=79.127.201.215:4345 , piece_id=GEW6KBEOWCDYMWWMGFINMELJWRW4TDWVSI42HK2IATDQRNCYU6YQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:50.875387857 (59h48m0.597950897s) [noise=true , address=77.90.31.12:4608 , piece_id=6LIGJWEHCKLJPRNIFJ4RLFGZ6DI6ETUKY2VGCOBAT3SKXQ7QB3SQ, seg_pos={0 0}, upload=57374]: Uploading piece... Jul 26 23:23:51.988632059 (59h48m1.711195099s) [upload=57374]: Next returned nil segment Jul 26 23:23:51.988657368 (59h48m1.711220408s) [upload=57374]: Waiting for error group managing segments... Jul 26 23:23:52.149552449 (59h48m1.872115489s) [noise=true , address=64.49.13.138:4483 , piece_id=MLLP3XDISSNKZX7W3VF7E3JPHHV2GONSXWHVMDNHREMLNLKQBZ4A, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:23:52.275333947 (59h48m1.997896987s) [noise=true , address=79.127.205.234:4591 , piece_id=MO5C6KOCIBGYLZJG4LRMZRHVXFHXZZJAT5X4E7HB4S2UMOPOAYVQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:52.288578267 (59h48m2.011141307s) [noise=true , address=79.127.205.246:4274 , piece_id=L3DV63UNNCT2M7WSSYLZGVHKUVW4I33OCHA2PHD3EE7ZQMILFZOA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:52.291147493 (59h48m2.013710543s) [noise=true , address=79.127.205.245:4223 , piece_id=EMR2ADKGWBRFFV3JJADP7RQKKJLF4C2JMX5FWA6GAAOCNHMU5SHA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:52.310706074 (59h48m2.033269114s) [noise=true , address=79.127.205.235:4690 , piece_id=EKY3LQAH2KZT4XQF5UPTTHK5DA2NJMAHEYVNTH74ESCI35EWBGLA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:53.713964672 (59h48m3.436527712s) [noise=true , address=79.127.219.47:4061 , piece_id=I64NW4VOJJAIQACPII6RRFA4WDX4Z6IUPSNBC3BCCYYJXTL547TA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:53.789594855 (59h48m3.512157895s) [noise=true , address=79.127.219.51:4210 , piece_id=3JF445FCNFCHZ2YY3NK5T4SXIAP22Z7FKE33IYWJYERYDNSRLUJQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:23:53.999308601 (59h48m3.721871651s) [noise=true , address=64.49.13.140:4508 , piece_id=MIANH7HBKK7MIWLRYW4G73X546NBPEBRDR4KSQYLUFZ4RM4O2F3A, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:23:57.155467977 (59h48m6.878031017s) [noise=true , address=45.91.116.14:4206 , piece_id=3ZDG5RILWVTXD3OIQOFBDPV5ETL52JGLIQA5ICKXOIFJNIRJWLNA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:24:00.233500552 (59h48m9.956063592s) [noise=true , address=64.49.13.140:4514 , piece_id=MRXU3VAKV5JH4GA7AVTQRTGPX2MVG6FHY6W5IRR5CLUSLBXSNSCQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:24:02.252832022 (59h48m11.975395083s) [noise=true , address=77.90.31.12:4608 , piece_id=6LIGJWEHCKLJPRNIFJ4RLFGZ6DI6ETUKY2VGCOBAT3SKXQ7QB3SQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:24:08.810674997 (59h48m18.533238037s) [noise=true , address=79.127.201.215:4345 , piece_id=GEW6KBEOWCDYMWWMGFINMELJWRW4TDWVSI42HK2IATDQRNCYU6YQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: <nil> Jul 26 23:24:49.783506276 (59h48m59.506069317s) [noise=true , address=104.36.82.55:4193 , piece_id=DGARUIVD3TQN5TR3TEIO3YJ5OUPJNSPWED77QNAGTHNTFUNGQZNA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:49.783590197 (59h48m59.506153237s) [noise=true , address=104.36.82.46:4507 , piece_id=XVKL6DI2P46BKN2Q6PC3WDMPVXNE2MN4SXDTWTFNLST7LAFOTV4Q, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:49.783719905 (59h48m59.506282935s) [noise=true , address=104.36.82.47:4549 , piece_id=5VHFQCLA6OLE5YDE5MJEGADQJTXLBICGEWE3RBKU7ERU5EHLRPPA, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: rpc: dial tcp 104.36.82.47:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:49.783850094 (59h48m59.506413134s) [noise=true , address=104.36.82.53:4142 , piece_id=BP3IKS4PSNGKLQXTXTPVIEACBBCC4QPALJ56VZMQ7CWNEVJLGVHQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:49.783915540 (59h48m59.50647858s) [noise=true , address=77.90.31.21:4887 , piece_id=66L5AAOBJ7MPQ4NGOZ75RAOFOQQRGKNY5E7NU6D3JSNA5TPVKYBQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:49.784015641 (59h48m59.506578681s) [noise=true , address=104.36.82.56:4231 , piece_id=LDPTNPZOX7E5KI5IW7SVWMG7NOJ34IS3A52LKVWXZ5JDTVBWIEAQ, seg_pos={0 0}, upload=57374]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:49.784060026 (59h48m59.506623066s) [seg_pos={0 0}, upload=57374]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:24:50.010858339 (59h48m59.733421379s) [upload=57374]: Done uploading segments... Jul 26 23:24:50.010890390 (59h48m59.73345343s) [upload=57374]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 1m0.229106629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [........X.X...........................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.236272768s FAIL (2025-07-26 23:24:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.236287806s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [......X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.229909672s FAIL (2025-07-26 23:24:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 7]: Begin upload segment... Jul 26 23:24:00.050313701 (59h48m9.772876741s) [seg_pos={0 0}, upload=57377]: Done begin upload segment. Jul 26 23:24:00.050336383 (59h48m9.772899424s) [noise=true , address=77.90.31.13:4626 , piece_id=AYSOT6UEWW3EXUBRA3466PTFQXBBMW4R7U77ZYW53MCRHIZXEBVQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.050378354 (59h48m9.772941384s) [noise=true , address=104.36.82.16:4293 , piece_id=WS2PESM3TO7C7ZZQGNYVLO6VITPQS43D3XX43KJRCZMS5ARLPDGA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.050759604 (59h48m9.773322644s) [noise=true , address=104.36.82.2:4731 , piece_id=ZCVXCIWGC3SPVVUDHUOPS5XJRAEP6MMID72EIAPY6OTCSNS4GFQQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.050962001 (59h48m9.773525341s) [noise=true , address=104.36.82.2:4730 , piece_id=BQFMXLSVMUEGZFVASY73ZMONPEPNJGBDID76Z433LG7N6C4FH4JA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.051125784 (59h48m9.773688824s) [noise=true , address=45.91.116.7:4011 , piece_id=ZSBLFC7PWYZVCSJJPMBVFF6DC4HOCWD6RWWLCJWSTK4KQPYZYDHA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.051488227 (59h48m9.774051257s) [noise=true , address=104.36.82.15:4259 , piece_id=5SZ2EBUBZ66G2C2TLEK6M7TZNBNKYK7M64RWIQWFW6ORAXQXZSUQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.051814231 (59h48m9.774377271s) [noise=true , address=77.90.31.10:4518 , piece_id=N3P6WX64SWV3OL5BJLI2NYUGB53IGS3IP4R3ODCHZID6XGUGMBJQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.052020817 (59h48m9.774583857s) [noise=true , address=104.36.82.14:4221 , piece_id=DUHWSZOFZ2TU3K2ODAPMDXBZMMNMMMHILHVLOIEPQPQLZ5AXNHEA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.052380585 (59h48m9.774943625s) [noise=true , address=45.91.116.10:4082 , piece_id=7II7FJJ5BYB6BFEMYDXGC2F6NUBHU5U2BQBIRPH36YHEDSZKZSOA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.052741365 (59h48m9.775304405s) [noise=true , address=104.36.82.5:4837 , piece_id=OPSKD4OH6YNXTOH7DO4DDZC3FVLOUQTFXOTWNV22HCOBKUF3TX5A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.052989741 (59h48m9.775552781s) [noise=true , address=45.91.116.11:4119 , piece_id=R56JPKEXDB4MVZHJUWKCNPAO6KHCTBP6AROWVJ2O3VPAL6NFP3BQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.053050016 (59h48m9.775613056s) [noise=true , address=104.36.82.5:4840 , piece_id=UT2FKSTGVMSURHR2P42YFQAK6NOXKXYEIUD6SKC2A4HIX5GCUFVQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.053291238 (59h48m9.775854278s) [noise=true , address=104.36.82.9:4007 , piece_id=7OZVQFIVH6QXZSZ6MQ7YIXBJYGEZLFMIV6QY6AYOUZDKZZWD63EQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.053517620 (59h48m9.776080661s) [noise=true , address=45.91.116.10:4083 , piece_id=O3POXPUOI4EGYUOJGOLMOK4JK3CBWGCFR6CZZDRFF6HNFXJ2TPDQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.053742221 (59h48m9.776305261s) [noise=true , address=104.36.82.9:4005 , piece_id=2P2DI2NVQ7AAXYDHJ5MKPCTJEBVRK5KYBF3RNDVCA4ZZVFOBYW5A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:00.054039530 (59h48m9.77660257s) [noise=true , address=104.36.82.8:4972 , piece_id=DAP7PZTYLGRBEUEEKVQK7OHFURDZJ23KUVTEXWWHLESP5RL5HC7A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:24:01.790236858 (59h48m11.512799908s) [upload=57377]: Next returned nil segment Jul 26 23:24:01.790269201 (59h48m11.512832241s) [upload=57377]: Waiting for error group managing segments... Jul 26 23:24:02.066337273 (59h48m11.788900313s) [noise=true , address=45.91.116.7:4011 , piece_id=ZSBLFC7PWYZVCSJJPMBVFF6DC4HOCWD6RWWLCJWSTK4KQPYZYDHA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:24:02.072075441 (59h48m11.794638481s) [noise=true , address=45.91.116.10:4082 , piece_id=7II7FJJ5BYB6BFEMYDXGC2F6NUBHU5U2BQBIRPH36YHEDSZKZSOA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:24:02.073871578 (59h48m11.796434618s) [noise=true , address=45.91.116.10:4083 , piece_id=O3POXPUOI4EGYUOJGOLMOK4JK3CBWGCFR6CZZDRFF6HNFXJ2TPDQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:24:02.279715659 (59h48m12.002278699s) [noise=true , address=45.91.116.11:4119 , piece_id=R56JPKEXDB4MVZHJUWKCNPAO6KHCTBP6AROWVJ2O3VPAL6NFP3BQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:24:03.526107665 (59h48m13.248670705s) [noise=true , address=77.90.31.13:4626 , piece_id=AYSOT6UEWW3EXUBRA3466PTFQXBBMW4R7U77ZYW53MCRHIZXEBVQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:24:29.181093146 (59h48m38.903656186s) [noise=true , address=104.36.82.14:4221 , piece_id=DUHWSZOFZ2TU3K2ODAPMDXBZMMNMMMHILHVLOIEPQPQLZ5AXNHEA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181139305 (59h48m38.903702345s) [noise=true , address=104.36.82.15:4259 , piece_id=5SZ2EBUBZ66G2C2TLEK6M7TZNBNKYK7M64RWIQWFW6ORAXQXZSUQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181172228 (59h48m38.903735268s) [noise=true , address=104.36.82.16:4293 , piece_id=WS2PESM3TO7C7ZZQGNYVLO6VITPQS43D3XX43KJRCZMS5ARLPDGA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:40614->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181178760 (59h48m38.9037418s) [noise=true , address=77.90.31.10:4518 , piece_id=N3P6WX64SWV3OL5BJLI2NYUGB53IGS3IP4R3ODCHZID6XGUGMBJQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181205130 (59h48m38.90376817s) [noise=true , address=104.36.82.2:4731 , piece_id=ZCVXCIWGC3SPVVUDHUOPS5XJRAEP6MMID72EIAPY6OTCSNS4GFQQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181215781 (59h48m38.903778821s) [noise=true , address=104.36.82.9:4007 , piece_id=7OZVQFIVH6QXZSZ6MQ7YIXBJYGEZLFMIV6QY6AYOUZDKZZWD63EQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181236952 (59h48m38.903799982s) [noise=true , address=104.36.82.2:4730 , piece_id=BQFMXLSVMUEGZFVASY73ZMONPEPNJGBDID76Z433LG7N6C4FH4JA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181245829 (59h48m38.903808859s) [noise=true , address=104.36.82.9:4005 , piece_id=2P2DI2NVQ7AAXYDHJ5MKPCTJEBVRK5KYBF3RNDVCA4ZZVFOBYW5A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181270887 (59h48m38.903833927s) [noise=true , address=104.36.82.5:4837 , piece_id=OPSKD4OH6YNXTOH7DO4DDZC3FVLOUQTFXOTWNV22HCOBKUF3TX5A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181278180 (59h48m38.90384122s) [noise=true , address=104.36.82.5:4840 , piece_id=UT2FKSTGVMSURHR2P42YFQAK6NOXKXYEIUD6SKC2A4HIX5GCUFVQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181281827 (59h48m38.903844867s) [noise=true , address=104.36.82.8:4972 , piece_id=DAP7PZTYLGRBEUEEKVQK7OHFURDZJ23KUVTEXWWHLESP5RL5HC7A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:54064->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:29.181304481 (59h48m38.903867521s) [seg_pos={0 0}, upload=57377]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:24:29.409588617 (59h48m39.132151647s) [upload=57377]: Done uploading segments... Jul 26 23:24:29.409617112 (59h48m39.132180152s) [upload=57377]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.229930732s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [......X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX..] duration: 31.04689236s FAIL (2025-07-26 23:24:23 UTC) orbiter warning: context deadline exceeded Duration: 31.046914402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......................................XXX.X..X.....XXXXX...X] duration: 30.001456737s FAIL (2025-07-26 23:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001481033s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [X........X.X...........................XXXXXXXXXXXXXXXXXXXXX] duration: 30.237441509s FAIL (2025-07-26 23:24:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.237454503s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.825984409s FAIL (2025-07-26 23:23:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.826013685s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233719931s FAIL (2025-07-26 23:23:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== noise=true , address=79.127.205.245:4252 , piece_id=ETI2L26HQI6FFTZVFRDRASPFVYPV73S7ZP6OQ6PQ5E6PT43DPBTA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457179577 (59h47m0.179742617s) [noise=true , address=79.127.205.234:4609 , piece_id=A4P53OX6JWPZUPAWAYJH5RZVKNDUSWPCNGZ4DE5SRBLVBUJYUSLA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457611865 (59h47m0.180174905s) [noise=true , address=79.127.205.233:4644 , piece_id=RH4Q4IUGPNAYALV2IMCT3JV7PA6VEQEO6UVHCQUEZ67KQ6U3SCXA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457825744 (59h47m0.180388784s) [noise=true , address=77.90.31.19:4850 , piece_id=5W6S6WGGSYXNFP7RDVWFAHYHS7EVL3EE6PWAJY5ZOJKY2R3ZGKKA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458023182 (59h47m0.180586222s) [noise=true , address=77.90.31.10:4543 , piece_id=OQGZGJHBI4XLMDCWOTTYYRCIYMERYYFTCWP4R3HHAOW2E5QM3ZZQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458130166 (59h47m0.180693206s) [noise=true , address=104.36.82.46:4501 , piece_id=CMQL7Z633XDFHTUORI4WAKZSKQKFP3NP426JJN5VTMMTPZC6JAFQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458309749 (59h47m0.18087279s) [noise=true , address=104.36.82.54:4168 , piece_id=BQ35DZDJRP3BYGXWTNB6OX6BSYJYSEH3OA3NEII32S7SVMFGN6AQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458559056 (59h47m0.181122096s) [noise=true , address=79.127.205.235:4666 , piece_id=JYBLHKDXS335W6XJ44Q3INQLL5JGLAVIPY2V4KOTMMOLOV3RS5TQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458600296 (59h47m0.181163336s) [noise=true , address=104.36.82.47:4538 , piece_id=33ZLU5ZWCTOMWTSM6XVXKG5JYFX3M4VMXPYXTREUANSLLOD3UM5Q, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458975032 (59h47m0.181538072s) [noise=true , address=104.36.82.56:4252 , piece_id=W3ST6N4BXGTG4WYMQD2QSRDXVZUDEE7DSBZJVF7ME7E43WIWEBHA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459217476 (59h47m0.181780516s) [noise=true , address=77.90.31.16:4761 , piece_id=MRRN4A7XRBOSQBPDJ6ZDXJWQZYG2OP7JSLGCTUDVAQXARLIFSY2A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459369087 (59h47m0.181932127s) [noise=true , address=77.90.31.18:4814 , piece_id=NNTX227RQG7SAXZDK6SS4FGVKVWQAD5NBU6RA4DZEHIMN3HZKBSQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459702925 (59h47m0.182265955s) [noise=true , address=45.91.116.12:4230 , piece_id=RL62YWLYNVJENMCOSXD4N47CJ46QROXWGBNEAD2RP27ZDO2PL7WQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460005374 (59h47m0.182568414s) [noise=true , address=104.36.82.55:4225 , piece_id=IA7W7XF7ACC35CVLPOIATVVS3FW3MCCHCSTHGKM4ZV6FN5CG5FTQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460516111 (59h47m0.183079151s) [noise=true , address=45.91.116.14:4222 , piece_id=72HI7P3FRZICTSKYWQU6HFS64Z6KL6TY3IBUXLRYSAOQWCCIX44A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460829801 (59h47m0.183392841s) [noise=true , address=77.90.31.11:4580 , piece_id=X6H5XSP4TDEFHDJVVVQZE2CAXAKWGPAFKTXBDSWACAYB52GERIOA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.461137901 (59h47m0.183700941s) [noise=true , address=104.36.82.52:4101 , piece_id=FE2KQOZ5Q2SHMDHIVGYN2KNTUD7CPSYXCC73JHVARTTW56DGPP2A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.461545511 (59h47m0.184108541s) [noise=true , address=45.91.116.9:4270 , piece_id=VQLSZYUJ35BRDMCYJ2WP3UW2JRLMZSJ4XVJIJILW4BYB2T6U6KRA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:51.571288125 (59h47m1.293851165s) [upload=57356]: Next returned nil segment Jul 26 23:22:51.571307603 (59h47m1.293870653s) [upload=57356]: Waiting for error group managing segments... Jul 26 23:22:51.851281448 (59h47m1.573844478s) [noise=true , address=79.127.205.233:4644 , piece_id=RH4Q4IUGPNAYALV2IMCT3JV7PA6VEQEO6UVHCQUEZ67KQ6U3SCXA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.864079465 (59h47m1.586642505s) [noise=true , address=79.127.205.234:4609 , piece_id=A4P53OX6JWPZUPAWAYJH5RZVKNDUSWPCNGZ4DE5SRBLVBUJYUSLA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.868213846 (59h47m1.590776886s) [noise=true , address=79.127.205.235:4666 , piece_id=JYBLHKDXS335W6XJ44Q3INQLL5JGLAVIPY2V4KOTMMOLOV3RS5TQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.882554543 (59h47m1.605117583s) [noise=true , address=79.127.205.245:4252 , piece_id=ETI2L26HQI6FFTZVFRDRASPFVYPV73S7ZP6OQ6PQ5E6PT43DPBTA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.331202453 (59h47m2.053765493s) [noise=true , address=45.91.116.12:4230 , piece_id=RL62YWLYNVJENMCOSXD4N47CJ46QROXWGBNEAD2RP27ZDO2PL7WQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.467399199 (59h47m2.189962239s) [noise=true , address=77.90.31.11:4580 , piece_id=X6H5XSP4TDEFHDJVVVQZE2CAXAKWGPAFKTXBDSWACAYB52GERIOA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.497447411 (59h47m2.220010441s) [noise=true , address=77.90.31.18:4814 , piece_id=NNTX227RQG7SAXZDK6SS4FGVKVWQAD5NBU6RA4DZEHIMN3HZKBSQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.889160627 (59h47m2.611723667s) [noise=true , address=45.91.116.9:4270 , piece_id=VQLSZYUJ35BRDMCYJ2WP3UW2JRLMZSJ4XVJIJILW4BYB2T6U6KRA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:53.886874268 (59h47m3.609437308s) [noise=true , address=45.91.116.14:4222 , piece_id=72HI7P3FRZICTSKYWQU6HFS64Z6KL6TY3IBUXLRYSAOQWCCIX44A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:23:12.575738489 (59h47m22.298301529s) [noise=true , address=77.90.31.19:4850 , piece_id=5W6S6WGGSYXNFP7RDVWFAHYHS7EVL3EE6PWAJY5ZOJKY2R3ZGKKA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:23:49.324977343 (59h47m59.047540383s) [noise=true , address=104.36.82.46:4501 , piece_id=CMQL7Z633XDFHTUORI4WAKZSKQKFP3NP426JJN5VTMMTPZC6JAFQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325025095 (59h47m59.047588125s) [noise=true , address=104.36.82.56:4252 , piece_id=W3ST6N4BXGTG4WYMQD2QSRDXVZUDEE7DSBZJVF7ME7E43WIWEBHA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: rpc: dial tcp 104.36.82.56:4252: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325057617 (59h47m59.047620657s) [noise=true , address=77.90.31.16:4761 , piece_id=MRRN4A7XRBOSQBPDJ6ZDXJWQZYG2OP7JSLGCTUDVAQXARLIFSY2A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 188.166.225.170:37552->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325073617 (59h47m59.047636647s) [noise=true , address=77.90.31.10:4543 , piece_id=OQGZGJHBI4XLMDCWOTTYYRCIYMERYYFTCWP4R3HHAOW2E5QM3ZZQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325096481 (59h47m59.047659521s) [noise=true , address=104.36.82.47:4538 , piece_id=33ZLU5ZWCTOMWTSM6XVXKG5JYFX3M4VMXPYXTREUANSLLOD3UM5Q, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325115728 (59h47m59.047678768s) [noise=true , address=104.36.82.52:4101 , piece_id=FE2KQOZ5Q2SHMDHIVGYN2KNTUD7CPSYXCC73JHVARTTW56DGPP2A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 188.166.225.170:34934->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325126338 (59h47m59.047689378s) [noise=true , address=104.36.82.55:4225 , piece_id=IA7W7XF7ACC35CVLPOIATVVS3FW3MCCHCSTHGKM4ZV6FN5CG5FTQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325131569 (59h47m59.047694599s) [noise=true , address=104.36.82.54:4168 , piece_id=BQ35DZDJRP3BYGXWTNB6OX6BSYJYSEH3OA3NEII32S7SVMFGN6AQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325161436 (59h47m59.047724476s) [seg_pos={0 0}, upload=57356]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:23:49.556768540 (59h47m59.27933158s) [upload=57356]: Done uploading segments... Jul 26 23:23:49.556797816 (59h47m59.279360856s) [upload=57356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.233731383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X........X.X...........................XXXXXXXXXXXXXXXXXXXX] duration: 30.231713323s FAIL (2025-07-26 23:23:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.231725266s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.......X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234611787s FAIL (2025-07-26 23:23:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ing upload Jul 26 23:22:57.001061404 (59h47m6.723624514s) [upload=57357]: Uploading segments... Jul 26 23:22:57.001349515 (59h47m6.723912585s) [upload=57357]: Got next segment. Inline: false Jul 26 23:22:57.001397407 (59h47m6.723960487s) [upload=57357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:57.001405772 (59h47m6.723968852s) [upload=57357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:57.239992641 (59h47m6.962555671s) [seg_pos={0 0}, upload=57357]: Begin upload segment... Jul 26 23:22:57.240036305 (59h47m6.962599345s) [seg_pos={0 0}, upload=57357]: Done begin upload segment. Jul 26 23:22:57.240057246 (59h47m6.962620276s) [noise=true , address=104.36.82.3:4765 , piece_id=ISSO3GMPZOCINXDFUC35BZD3SSHLEYIKG2WFLSTETRBHFWQARHFA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240114756 (59h47m6.962677796s) [noise=true , address=45.91.116.7:4011 , piece_id=NAORINYA6WT5JNHQIOXP6I5SBG4CRPPRDRBTVQVOVE6XZPFJM7FA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240465437 (59h47m6.963028467s) [noise=true , address=104.36.82.2:4729 , piece_id=6GB5QRQ3R7PF5KMWJO4IOYP32WEGDI6KKDG35IYQQXP4DFKHJCKQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240731295 (59h47m6.963294335s) [noise=true , address=45.91.116.10:4083 , piece_id=MIMPJTD7WHDGTW4MS4XP3HY3SGLGKWF4OICZBU64U5M63P4DC2NQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240769839 (59h47m6.963332879s) [noise=true , address=104.36.82.10:4078 , piece_id=TRH6SMMVIMOEHYAMEWSOUUCLRD3YDWZ7AUUTWRJSXUDR3WU3M6AA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240995010 (59h47m6.96355805s) [noise=true , address=104.36.82.6:4873 , piece_id=3TYPHXHEXZAJFTNZZO2Z7FHA5LTYYQGGDOCGNRLK4RJULLLCJGMQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241130880 (59h47m6.96369392s) [noise=true , address=77.90.31.15:4697 , piece_id=TNRCHE4MVDYP2K4PCXGFXJ4BT7TQLGOO3DQBRUY6DYMFBQK3K75A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241254456 (59h47m6.963817496s) [noise=true , address=77.90.31.18:4805 , piece_id=AP2IJBH26NFU7JQITMNVOQTQKBWEGMPB2FCIB4RJJAFNIVSRHZYQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241482341 (59h47m6.964045382s) [noise=true , address=45.91.116.12:4225 , piece_id=R2CJ5CHOJEU2R4YQHBLMSZSJME5XKIV7S4JTEJLFNON4RPUDOCFQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241488133 (59h47m6.964051173s) [noise=true , address=104.36.82.7:4909 , piece_id=BXKNHFCLRR6RFWGIIICZ3ABYJTGZR5EMPRUCLTTR7O63Y4XRUX6A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241727270 (59h47m6.96429031s) [noise=true , address=77.90.31.10:4517 , piece_id=4NY2T5BT24RH3YY5EDXTWBXLAW5DK5FQ2SUI473B4ZCKFHDJRWTA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241780803 (59h47m6.964343843s) [noise=true , address=104.36.82.15:4259 , piece_id=OK23G7EAA4TRNKRWT2723S3QDLW7Y3H6JADC7N4PN6EEZEKSAADQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241992398 (59h47m6.964555428s) [noise=true , address=104.36.82.2:4732 , piece_id=U7RZRKKTE2A3YT2NAXHZPGGGRXADKBJNRMY7JDR5VMDR4A27TCNQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242236435 (59h47m6.964799475s) [noise=true , address=45.91.116.6:4975 , piece_id=2SENOK7EWBSCJXPAAHS6D4L62DGPJG67WHPGYP64ZM4PLSHCLYUA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242446337 (59h47m6.965009367s) [noise=true , address=77.90.31.19:4842 , piece_id=DAERSSUY7VX5UXNWQR5HNT2U22WMWNNMEKGJ4UJZXVSRP7GWZ5YA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242638004 (59h47m6.965201044s) [noise=true , address=104.36.82.5:4838 , piece_id=XBFMLFEPHOU56S5QATKOYJO6SOOHGW3IHM3VODDIBNB42G4XRUJA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:58.981229929 (59h47m8.703792959s) [upload=57357]: Next returned nil segment Jul 26 23:22:58.981251962 (59h47m8.703815002s) [upload=57357]: Waiting for error group managing segments... Jul 26 23:22:59.258281663 (59h47m8.980844703s) [noise=true , address=45.91.116.10:4083 , piece_id=MIMPJTD7WHDGTW4MS4XP3HY3SGLGKWF4OICZBU64U5M63P4DC2NQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.274332333 (59h47m8.996895373s) [noise=true , address=45.91.116.7:4011 , piece_id=NAORINYA6WT5JNHQIOXP6I5SBG4CRPPRDRBTVQVOVE6XZPFJM7FA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.337077664 (59h47m9.059640694s) [noise=true , address=45.91.116.12:4225 , piece_id=R2CJ5CHOJEU2R4YQHBLMSZSJME5XKIV7S4JTEJLFNON4RPUDOCFQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.365926510 (59h47m9.08848955s) [noise=true , address=45.91.116.6:4975 , piece_id=2SENOK7EWBSCJXPAAHS6D4L62DGPJG67WHPGYP64ZM4PLSHCLYUA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.456581532 (59h47m9.179144582s) [noise=true , address=77.90.31.15:4697 , piece_id=TNRCHE4MVDYP2K4PCXGFXJ4BT7TQLGOO3DQBRUY6DYMFBQK3K75A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:00.587592219 (59h47m10.310155259s) [noise=true , address=77.90.31.18:4805 , piece_id=AP2IJBH26NFU7JQITMNVOQTQKBWEGMPB2FCIB4RJJAFNIVSRHZYQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:01.390453636 (59h47m11.113016676s) [noise=true , address=77.90.31.19:4842 , piece_id=DAERSSUY7VX5UXNWQR5HNT2U22WMWNNMEKGJ4UJZXVSRP7GWZ5YA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:26.338955588 (59h47m36.061518628s) [noise=true , address=104.36.82.10:4078 , piece_id=TRH6SMMVIMOEHYAMEWSOUUCLRD3YDWZ7AUUTWRJSXUDR3WU3M6AA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.338997900 (59h47m36.06156094s) [noise=true , address=104.36.82.3:4765 , piece_id=ISSO3GMPZOCINXDFUC35BZD3SSHLEYIKG2WFLSTETRBHFWQARHFA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339028919 (59h47m36.061591949s) [noise=true , address=104.36.82.15:4259 , piece_id=OK23G7EAA4TRNKRWT2723S3QDLW7Y3H6JADC7N4PN6EEZEKSAADQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339059547 (59h47m36.061622587s) [noise=true , address=104.36.82.2:4729 , piece_id=6GB5QRQ3R7PF5KMWJO4IOYP32WEGDI6KKDG35IYQQXP4DFKHJCKQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339091448 (59h47m36.061654478s) [noise=true , address=104.36.82.6:4873 , piece_id=3TYPHXHEXZAJFTNZZO2Z7FHA5LTYYQGGDOCGNRLK4RJULLLCJGMQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339118640 (59h47m36.06168168s) [noise=true , address=104.36.82.7:4909 , piece_id=BXKNHFCLRR6RFWGIIICZ3ABYJTGZR5EMPRUCLTTR7O63Y4XRUX6A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339136915 (59h47m36.061699956s) [noise=true , address=77.90.31.10:4517 , piece_id=4NY2T5BT24RH3YY5EDXTWBXLAW5DK5FQ2SUI473B4ZCKFHDJRWTA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339154428 (59h47m36.061717468s) [noise=true , address=104.36.82.2:4732 , piece_id=U7RZRKKTE2A3YT2NAXHZPGGGRXADKBJNRMY7JDR5VMDR4A27TCNQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339180538 (59h47m36.061743578s) [noise=true , address=104.36.82.5:4838 , piece_id=XBFMLFEPHOU56S5QATKOYJO6SOOHGW3IHM3VODDIBNB42G4XRUJA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:37568->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339209243 (59h47m36.061772293s) [seg_pos={0 0}, upload=57357]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:23:26.572151393 (59h47m36.294714433s) [upload=57357]: Done uploading segments... Jul 26 23:23:26.572210005 (59h47m36.294773085s) [upload=57357]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.234637617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.......X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.] duration: 43.677234768s FAIL (2025-07-26 23:23:25 UTC) orbiter warning: context deadline exceeded Duration: 43.677244666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.866306812s FAIL (2025-07-26 23:23:14 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.866335265s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................................XXX.X..X.....XXXXX...] duration: 14.371021092s FAIL (2025-07-26 23:23:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.371041701s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X........X.X...........................XXXXXXXXXXXXXXXXXXX] duration: 30.2392811s FAIL (2025-07-26 23:23:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239294816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245181498s FAIL (2025-07-26 23:22:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 6E7IIGSVI6QVXP7L3XMGWIYBBFQST7HKK6O73QJ2XBQ3LFVJMQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946096069 (59h45m59.668659109s) [noise=true , address=104.36.82.52:4107 , piece_id=Z5LAQNDCZY4DSJYHBY7BFCNACMLKVQWVRMBC2REOPZ7DP32FBKIQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946254011 (59h45m59.668817051s) [noise=true , address=64.49.13.140:4508 , piece_id=OFRK3PG7TDAQDUQXXKNWDFKDGEN33HTCCCSJZP6WLVIZJCUSPLTA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946371356 (59h45m59.668934396s) [noise=true , address=77.90.31.19:4875 , piece_id=Z2FCMAFAYVA2H4XVXKIFJ3A6IG7OFRW42CUC4GBV6TJKDZHLLLRA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946646292 (59h45m59.669209332s) [noise=true , address=77.90.31.17:4791 , piece_id=UC6Q6TP5M2T5JH34XXGZEM3GAUOPFLZXUV25SNEZNZB7TV4FLLCA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946899796 (59h45m59.669462826s) [noise=true , address=104.36.82.12:4173 , piece_id=UV4ZD4DRSIY27IIV5VYR3P67KA65ZDFS5AV2WAS6WIVJ4LLA2BLA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947131671 (59h45m59.669694711s) [noise=true , address=104.36.82.46:4511 , piece_id=4ITN25O7TMLSQSQKNS76KHAKEZGFZSZDM5XVFCXJVDCPM6N2O37A, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947454017 (59h45m59.670017057s) [noise=true , address=79.127.205.233:4649 , piece_id=5ZOLYN57XWVFDROD3647N2AQXWLSZME3ANICO6ORHDNYXNS5OSVQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947746757 (59h45m59.670309797s) [noise=true , address=45.91.116.13:4175 , piece_id=36ONATBRVOPS4JILW3GRKPEKDGEYD2ZFWSA2JNKDOL52CRQLWO5A, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948009058 (59h45m59.670572088s) [noise=true , address=79.127.205.246:4260 , piece_id=6FWM6RRLYQ43MUPCQQWTTYXJLJ2EJYT7PAVEQTHMI4AIQHWOYVRQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948335914 (59h45m59.670898954s) [noise=true , address=104.36.82.53:4150 , piece_id=2JP7PHJNM33ISTAVBRRIXRUWW7UTNMFABZAT3WU3QEM6YNFLGGPQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948599649 (59h45m59.671162689s) [noise=true , address=79.127.205.235:4666 , piece_id=E4KJFEUHYNAAAQS7TWGPQLSGXAEG6OKXFIWXYYVNEDMKYRTPW5DQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948845239 (59h45m59.671408279s) [noise=true , address=104.36.82.55:4210 , piece_id=2DM6B7VNRHVIMS6PQQQZ546NJQ6VCJ4W6UBBG6TVRP6XEBOVFAAQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949103392 (59h45m59.671666432s) [noise=true , address=79.127.220.101:4048 , piece_id=22D5A6NLUOAL2D5ZINJFLQVE5XNLCBPJNFKFQPLCYDEHCI6UM2EA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949420981 (59h45m59.671984021s) [noise=true , address=104.36.82.47:4526 , piece_id=BXABWY7KOKW7FMEPJCTB7PG3A2OXHVW6R7KF7DVG7WMXXKIJNLWA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949559315 (59h45m59.672122355s) [noise=true , address=64.49.13.140:4514 , piece_id=XB2CD4WKSQ6NG5UKYCRYBWO7O5ANJB3HQCOLQCGB52TBRWFBUKIA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949844120 (59h45m59.67240715s) [noise=true , address=104.36.82.54:4183 , piece_id=NS6TTWKDZ2USNM7QODIOI3NOWB2FKSPCEZQMCXLO5ESRL6NFYZPA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:51.056692928 (59h46m0.779255958s) [upload=57340]: Next returned nil segment Jul 26 23:21:51.056710000 (59h46m0.77927304s) [upload=57340]: Waiting for error group managing segments... Jul 26 23:21:51.353937997 (59h46m1.076501028s) [noise=true , address=79.127.205.246:4260 , piece_id=6FWM6RRLYQ43MUPCQQWTTYXJLJ2EJYT7PAVEQTHMI4AIQHWOYVRQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.361104319 (59h46m1.083667359s) [noise=true , address=79.127.205.235:4666 , piece_id=E4KJFEUHYNAAAQS7TWGPQLSGXAEG6OKXFIWXYYVNEDMKYRTPW5DQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.385354617 (59h46m1.107917657s) [noise=true , address=79.127.205.233:4649 , piece_id=5ZOLYN57XWVFDROD3647N2AQXWLSZME3ANICO6ORHDNYXNS5OSVQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.495253076 (59h46m1.217816116s) [noise=true , address=79.127.205.234:4597 , piece_id=QTQVIJEN7DYU7P5JR35H62GBA7JK2LZABMBJ437HVIQAZWZBR4HQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:55.807535997 (59h46m5.530099037s) [noise=true , address=79.127.220.101:4048 , piece_id=22D5A6NLUOAL2D5ZINJFLQVE5XNLCBPJNFKFQPLCYDEHCI6UM2EA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:59.641000414 (59h46m9.363563444s) [noise=true , address=64.49.13.140:4514 , piece_id=XB2CD4WKSQ6NG5UKYCRYBWO7O5ANJB3HQCOLQCGB52TBRWFBUKIA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:00.137462077 (59h46m9.860025107s) [noise=true , address=64.49.13.140:4508 , piece_id=OFRK3PG7TDAQDUQXXKNWDFKDGEN33HTCCCSJZP6WLVIZJCUSPLTA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:05.935782658 (59h46m15.658345698s) [noise=true , address=77.90.31.19:4875 , piece_id=Z2FCMAFAYVA2H4XVXKIFJ3A6IG7OFRW42CUC4GBV6TJKDZHLLLRA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:10.633348902 (59h46m20.355911943s) [noise=true , address=45.91.116.13:4175 , piece_id=36ONATBRVOPS4JILW3GRKPEKDGEYD2ZFWSA2JNKDOL52CRQLWO5A, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:23.241788007 (59h46m32.964351047s) [noise=true , address=79.127.201.215:4354 , piece_id=RL6E7IIGSVI6QVXP7L3XMGWIYBBFQST7HKK6O73QJ2XBQ3LFVJMQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:48.828518062 (59h46m58.551081102s) [noise=true , address=104.36.82.52:4107 , piece_id=Z5LAQNDCZY4DSJYHBY7BFCNACMLKVQWVRMBC2REOPZ7DP32FBKIQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 188.166.225.170:59748->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828651207 (59h46m58.551214247s) [noise=true , address=104.36.82.53:4150 , piece_id=2JP7PHJNM33ISTAVBRRIXRUWW7UTNMFABZAT3WU3QEM6YNFLGGPQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: write tcp 188.166.225.170:43584->104.36.82.53:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828703316 (59h46m58.551266346s) [noise=true , address=104.36.82.47:4526 , piece_id=BXABWY7KOKW7FMEPJCTB7PG3A2OXHVW6R7KF7DVG7WMXXKIJNLWA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:55928->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828776877 (59h46m58.551339917s) [noise=true , address=104.36.82.54:4183 , piece_id=NS6TTWKDZ2USNM7QODIOI3NOWB2FKSPCEZQMCXLO5ESRL6NFYZPA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:33370->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828815892 (59h46m58.551378922s) [noise=true , address=104.36.82.55:4210 , piece_id=2DM6B7VNRHVIMS6PQQQZ546NJQ6VCJ4W6UBBG6TVRP6XEBOVFAAQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 188.166.225.170:59992->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828855939 (59h46m58.551418969s) [noise=true , address=77.90.31.17:4791 , piece_id=UC6Q6TP5M2T5JH34XXGZEM3GAUOPFLZXUV25SNEZNZB7TV4FLLCA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: write tcp 188.166.225.170:32836->77.90.31.17:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828881958 (59h46m58.551445008s) [noise=true , address=104.36.82.12:4173 , piece_id=UV4ZD4DRSIY27IIV5VYR3P67KA65ZDFS5AV2WAS6WIVJ4LLA2BLA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 188.166.225.170:56022->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828918348 (59h46m58.551481388s) [noise=true , address=104.36.82.46:4511 , piece_id=4ITN25O7TMLSQSQKNS76KHAKEZGFZSZDM5XVFCXJVDCPM6N2O37A, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: write tcp 188.166.225.170:55300->104.36.82.46:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828946632 (59h46m58.551509662s) [seg_pos={0 0}, upload=57340]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:22:49.072235989 (59h46m58.794799029s) [upload=57340]: Done uploading segments... Jul 26 23:22:49.072259785 (59h46m58.794822825s) [upload=57340]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.245193221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XXX.X..X.....XXXXX..] duration: 11.692395454s FAIL (2025-07-26 23:22:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.692421493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX] duration: 1m0.001397937s FAIL (2025-07-26 23:22:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001410541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [........X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXX] duration: 30.269336184s FAIL (2025-07-26 23:22:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 9h46m11.265765536s) [upload=57344]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:01.793693011 (59h46m11.516256051s) [seg_pos={0 0}, upload=57344]: Begin upload segment... Jul 26 23:22:01.793746623 (59h46m11.516309663s) [seg_pos={0 0}, upload=57344]: Done begin upload segment. Jul 26 23:22:01.793782922 (59h46m11.516345992s) [noise=true , address=104.36.82.9:4006 , piece_id=YD765OAN7LU7DJATVJ44KU343OWU34UXE6KEMY6VMDK3IVS33NXQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.793827268 (59h46m11.516390308s) [noise=true , address=104.36.82.10:4079 , piece_id=HR4KE5RWXWM6DMTTEXAKXPEW4TXL4KZAQWYAN7USZ5N6JOJAHODQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794225018 (59h46m11.516788058s) [noise=true , address=104.36.82.13:4185 , piece_id=S3X72RTESXC342YZKXYEUMED4R6FFTOIX5A33KNA3ENS54TVCGEQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794300914 (59h46m11.516863954s) [noise=true , address=77.90.31.12:4589 , piece_id=OYGWROTKXGEP2EY46MSL7LK73K6JGP2P7KM7S2AD2ZXTUE7TKFHQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794503521 (59h46m11.517066571s) [noise=true , address=104.36.82.10:4077 , piece_id=U34TRM6MAQJTCFWPMQZWI3AZ6A2ROV7BOVQPN2I6RFJMQY2YPR2Q, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794741256 (59h46m11.517304296s) [noise=true , address=104.36.82.7:4910 , piece_id=ZLX5RCPRSVSUOBZ7WGCC3NHUDIDWXZKCVVDAAK7MUA6VWMHR33JQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795007155 (59h46m11.517570195s) [noise=true , address=104.36.82.14:4221 , piece_id=T322WMUJHZ5OIO4ZL2A33HMGB7AHPDRQ76SLARHHADAF3U6WOH3A, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795245370 (59h46m11.51780841s) [noise=true , address=104.36.82.13:4187 , piece_id=HMYYVZHWYZ4LN6DS7LMPW7FUBGKQKQZOOPVYQXENGLQHL32HA4AQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795524244 (59h46m11.518087284s) [noise=true , address=45.91.116.13:4153 , piece_id=URITHVCGSEVEDHJS2AYVJQARMA43TFGJ4HP7AFU2EYEK3WZ6J5GQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795759083 (59h46m11.518322123s) [noise=true , address=104.36.82.2:4729 , piece_id=AMDDQ3BHDJLFL5ZK7BOAUUNTBGLTHH73F5THCJENBLJQGFBA2FAQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795840810 (59h46m11.51840386s) [noise=true , address=104.36.82.4:4803 , piece_id=RIRXNNPHUBC36SEECE2TNW2OIRJNMLQHGV6QJEC7LES73GYQ5CVA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796097180 (59h46m11.51866022s) [noise=true , address=45.91.116.10:4083 , piece_id=WWQETFRHMNN2NGR2MUP74UR766JEATD2SZQ7EQDLWMETWCJPKCSA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796363340 (59h46m11.51892638s) [noise=true , address=104.36.82.5:4840 , piece_id=G6IHATPVRAFEHTIKTH3ZJDD7BGGM7PD42HYYXKQN2L6TAF6TY32Q, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796475424 (59h46m11.519038464s) [noise=true , address=45.91.116.7:4009 , piece_id=RYSA7RUBXXUAXE6TPSPK66UTGDC5PM2RVGEE7RYRTTGHPERDIMPQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796714472 (59h46m11.519277512s) [noise=true , address=77.90.31.14:4661 , piece_id=H3H5UMWXQFE4Q3DV74WZCVV4W5VZ3PXBICSVFEJG3S6AGAZ76YKQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796924333 (59h46m11.519487373s) [noise=true , address=104.36.82.7:4912 , piece_id=YF3FFPLFZDPPSEMCMM4N5UHT3LRIQ4RBDCMNFSS3PI7QKIB76EJA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:03.506866463 (59h46m13.229429504s) [upload=57344]: Next returned nil segment Jul 26 23:22:03.506879498 (59h46m13.229442538s) [upload=57344]: Waiting for error group managing segments... Jul 26 23:22:03.839512867 (59h46m13.562075907s) [noise=true , address=45.91.116.10:4083 , piece_id=WWQETFRHMNN2NGR2MUP74UR766JEATD2SZQ7EQDLWMETWCJPKCSA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.859393384 (59h46m13.581956424s) [noise=true , address=104.36.82.5:4840 , piece_id=G6IHATPVRAFEHTIKTH3ZJDD7BGGM7PD42HYYXKQN2L6TAF6TY32Q, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.895081837 (59h46m13.617644897s) [noise=true , address=45.91.116.7:4009 , piece_id=RYSA7RUBXXUAXE6TPSPK66UTGDC5PM2RVGEE7RYRTTGHPERDIMPQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.899200437 (59h46m13.621763517s) [noise=true , address=77.90.31.14:4661 , piece_id=H3H5UMWXQFE4Q3DV74WZCVV4W5VZ3PXBICSVFEJG3S6AGAZ76YKQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.930337241 (59h46m13.652900281s) [noise=true , address=77.90.31.12:4589 , piece_id=OYGWROTKXGEP2EY46MSL7LK73K6JGP2P7KM7S2AD2ZXTUE7TKFHQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.949333096 (59h46m13.671896136s) [noise=true , address=45.91.116.13:4153 , piece_id=URITHVCGSEVEDHJS2AYVJQARMA43TFGJ4HP7AFU2EYEK3WZ6J5GQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:30.898358548 (59h46m40.620921588s) [noise=true , address=104.36.82.10:4079 , piece_id=HR4KE5RWXWM6DMTTEXAKXPEW4TXL4KZAQWYAN7USZ5N6JOJAHODQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898404034 (59h46m40.620967074s) [noise=true , address=104.36.82.13:4185 , piece_id=S3X72RTESXC342YZKXYEUMED4R6FFTOIX5A33KNA3ENS54TVCGEQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:36952->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898438451 (59h46m40.621001491s) [noise=true , address=104.36.82.7:4910 , piece_id=ZLX5RCPRSVSUOBZ7WGCC3NHUDIDWXZKCVVDAAK7MUA6VWMHR33JQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898474509 (59h46m40.621037549s) [noise=true , address=104.36.82.14:4221 , piece_id=T322WMUJHZ5OIO4ZL2A33HMGB7AHPDRQ76SLARHHADAF3U6WOH3A, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:33394->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898485149 (59h46m40.621048179s) [noise=true , address=104.36.82.13:4187 , piece_id=HMYYVZHWYZ4LN6DS7LMPW7FUBGKQKQZOOPVYQXENGLQHL32HA4AQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898512682 (59h46m40.621075722s) [noise=true , address=104.36.82.7:4912 , piece_id=YF3FFPLFZDPPSEMCMM4N5UHT3LRIQ4RBDCMNFSS3PI7QKIB76EJA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898516980 (59h46m40.62108001s) [noise=true , address=104.36.82.10:4077 , piece_id=U34TRM6MAQJTCFWPMQZWI3AZ6A2ROV7BOVQPN2I6RFJMQY2YPR2Q, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898548862 (59h46m40.621111902s) [noise=true , address=104.36.82.9:4006 , piece_id=YD765OAN7LU7DJATVJ44KU343OWU34UXE6KEMY6VMDK3IVS33NXQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898559352 (59h46m40.621122392s) [noise=true , address=104.36.82.2:4729 , piece_id=AMDDQ3BHDJLFL5ZK7BOAUUNTBGLTHH73F5THCJENBLJQGFBA2FAQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898580753 (59h46m40.621143793s) [noise=true , address=104.36.82.4:4803 , piece_id=RIRXNNPHUBC36SEECE2TNW2OIRJNMLQHGV6QJEC7LES73GYQ5CVA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 188.166.225.170:49390->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898605099 (59h46m40.6211682s) [seg_pos={0 0}, upload=57344]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:22:31.166289399 (59h46m40.888852439s) [upload=57344]: Done uploading segments... Jul 26 23:22:31.166316231 (59h46m40.888879271s) [upload=57344]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.269350773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X........X.X...........................XXXXXXXXXXXXXXXXXX] duration: 30.247709882s FAIL (2025-07-26 23:22:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.247741071s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.012537356s FAIL (2025-07-26 23:22:24 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.012564389s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........X.........X...........X.......XXXXXXXXXXXXXXXXXXXXX] duration: 30.28566576s FAIL (2025-07-26 23:22:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== starting upload Jul 26 23:21:31.079201416 (59h45m40.801764486s) [upload=57334]: Uploading segments... Jul 26 23:21:31.079549612 (59h45m40.802112672s) [upload=57334]: Got next segment. Inline: false Jul 26 23:21:31.079599757 (59h45m40.802162827s) [upload=57334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:21:31.079607994 (59h45m40.802171064s) [upload=57334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:21:31.315397450 (59h45m41.03796049s) [seg_pos={0 0}, upload=57334]: Begin upload segment... Jul 26 23:21:31.315446834 (59h45m41.038009874s) [seg_pos={0 0}, upload=57334]: Done begin upload segment. Jul 26 23:21:31.315466151 (59h45m41.038029191s) [noise=true , address=77.90.31.19:4842 , piece_id=6U6HGX3ZIMQNWRAMOJGCDXSI6UR2C3XLOBQ47IRMO22NFIW6FGFQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.315475268 (59h45m41.038038308s) [noise=true , address=45.91.116.5:4297 , piece_id=D344A2TZZZ6GCOKYHCSLEYWK4GXOSZINHLZ3SGOXYIF65JRN4BSA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.315796934 (59h45m41.038359974s) [noise=true , address=104.36.82.12:4149 , piece_id=IG4VMFEDBIOVAXJKNGM76AEOSRIDASTZ63AFR5SD3ZPVMMYBLQPQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316041151 (59h45m41.038604192s) [noise=true , address=104.36.82.2:4729 , piece_id=6FUK3H5KAVWFFFZQQCQJVR3IBXYLEVWPKMCPMG7LNKQK6HUT333Q, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316315496 (59h45m41.038878536s) [noise=true , address=104.36.82.7:4910 , piece_id=WTOXS2R5UINBYNC6RL3CPCIJ5JVGOZHISLKKPSM7ISHKT44S36BA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316553933 (59h45m41.039116963s) [noise=true , address=77.90.31.14:4661 , piece_id=Q4MKNDWIPQBIRFUFBDRWZRFLEPHZRML5JOWVQJLEYIG2S2DDYCWA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316786688 (59h45m41.039349728s) [noise=true , address=104.36.82.9:4007 , piece_id=JZM6F4MPJSO4TOGKUENSMJWPIKWO72Y2RLJM6R6DCN5XKOEGCHCA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.317557232 (59h45m41.040120272s) [noise=true , address=45.91.116.9:4261 , piece_id=MGKCTJ6ZIIGXEJOYAEEYFZCMV2FBZVMQ33HUSAV3KZXHKZXSJ55A, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.317973308 (59h45m41.040536348s) [noise=true , address=104.36.82.5:4837 , piece_id=OZ5KWYRELI74XCVPZ6VHSXH2GHH5X6QSZWS6OBUQDS6UZRLMQXSQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.318438448 (59h45m41.041001488s) [noise=true , address=45.91.116.14:4191 , piece_id=HGLJN2HB7IEYSBNSYZ5T2TKR4SI5ZJ5OELRJXW3HA4R3XHSWGSZA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.318793447 (59h45m41.041356487s) [noise=true , address=104.36.82.10:4077 , piece_id=H6EHHMERH5Q6D7FRUOOE26YWFA5I4YA3KZDG2XW4JVWWSKH2UWEQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319037494 (59h45m41.041600534s) [noise=true , address=104.36.82.14:4222 , piece_id=VEUQZ32IOANF4ETMPAPOF5ZEERJMBF6DSYIBIIE6OHBAPIHGVBMA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319327840 (59h45m41.04189088s) [noise=true , address=104.36.82.9:4005 , piece_id=EERIWAPGKC7TE7BO3OI3F7UPIVJZWYGRTCGHB3DBSDWS6HOJQYVQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319558030 (59h45m41.04212106s) [noise=true , address=45.91.116.13:4154 , piece_id=O2PJG7XBIRJHVT6JMI3KGVF5666GYNIB3KD7YDD25D5H6ZMLON7Q, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319779394 (59h45m41.042342434s) [noise=true , address=104.36.82.13:4186 , piece_id=5GY2XWA7XVCW67BLQFLYFEMJKWNNARVAHXGPNAI6IP67V6QZQMAA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319993223 (59h45m41.042556263s) [noise=true , address=104.36.82.4:4802 , piece_id=7ZDQEMVP7H3SAI425NWBF2EM73HM43UHSRUWECTSSHA5ICV7OQCA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:33.025563649 (59h45m42.748126689s) [upload=57334]: Next returned nil segment Jul 26 23:21:33.025582465 (59h45m42.748145505s) [upload=57334]: Waiting for error group managing segments... Jul 26 23:21:33.340809481 (59h45m43.063372521s) [noise=true , address=77.90.31.14:4661 , piece_id=Q4MKNDWIPQBIRFUFBDRWZRFLEPHZRML5JOWVQJLEYIG2S2DDYCWA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.409544172 (59h45m43.132107202s) [noise=true , address=45.91.116.13:4154 , piece_id=O2PJG7XBIRJHVT6JMI3KGVF5666GYNIB3KD7YDD25D5H6ZMLON7Q, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.416314225 (59h45m43.138877255s) [noise=true , address=104.36.82.5:4837 , piece_id=OZ5KWYRELI74XCVPZ6VHSXH2GHH5X6QSZWS6OBUQDS6UZRLMQXSQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.436711050 (59h45m43.15927409s) [noise=true , address=45.91.116.5:4297 , piece_id=D344A2TZZZ6GCOKYHCSLEYWK4GXOSZINHLZ3SGOXYIF65JRN4BSA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.497740649 (59h45m43.220303689s) [noise=true , address=45.91.116.14:4191 , piece_id=HGLJN2HB7IEYSBNSYZ5T2TKR4SI5ZJ5OELRJXW3HA4R3XHSWGSZA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.843048748 (59h45m43.565611788s) [noise=true , address=45.91.116.9:4261 , piece_id=MGKCTJ6ZIIGXEJOYAEEYFZCMV2FBZVMQ33HUSAV3KZXHKZXSJ55A, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:35.443196869 (59h45m45.165759909s) [noise=true , address=77.90.31.19:4842 , piece_id=6U6HGX3ZIMQNWRAMOJGCDXSI6UR2C3XLOBQ47IRMO22NFIW6FGFQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:22:00.377926558 (59h46m10.100489598s) [noise=true , address=104.36.82.12:4149 , piece_id=IG4VMFEDBIOVAXJKNGM76AEOSRIDASTZ63AFR5SD3ZPVMMYBLQPQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.377988256 (59h46m10.100551296s) [noise=true , address=104.36.82.2:4729 , piece_id=6FUK3H5KAVWFFFZQQCQJVR3IBXYLEVWPKMCPMG7LNKQK6HUT333Q, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378051046 (59h46m10.100614086s) [noise=true , address=104.36.82.7:4910 , piece_id=WTOXS2R5UINBYNC6RL3CPCIJ5JVGOZHISLKKPSM7ISHKT44S36BA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378113666 (59h46m10.100676706s) [noise=true , address=104.36.82.9:4007 , piece_id=JZM6F4MPJSO4TOGKUENSMJWPIKWO72Y2RLJM6R6DCN5XKOEGCHCA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378190132 (59h46m10.100753172s) [noise=true , address=104.36.82.10:4077 , piece_id=H6EHHMERH5Q6D7FRUOOE26YWFA5I4YA3KZDG2XW4JVWWSKH2UWEQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378225520 (59h46m10.10078856s) [noise=true , address=104.36.82.4:4802 , piece_id=7ZDQEMVP7H3SAI425NWBF2EM73HM43UHSRUWECTSSHA5ICV7OQCA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378296996 (59h46m10.100860037s) [noise=true , address=104.36.82.14:4222 , piece_id=VEUQZ32IOANF4ETMPAPOF5ZEERJMBF6DSYIBIIE6OHBAPIHGVBMA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378355839 (59h46m10.100918879s) [noise=true , address=104.36.82.13:4186 , piece_id=5GY2XWA7XVCW67BLQFLYFEMJKWNNARVAHXGPNAI6IP67V6QZQMAA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378413600 (59h46m10.10097664s) [noise=true , address=104.36.82.9:4005 , piece_id=EERIWAPGKC7TE7BO3OI3F7UPIVJZWYGRTCGHB3DBSDWS6HOJQYVQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:57714->104.36.82.9:4005: use of closed network connection; context deadline exceeded Jul 26 23:22:00.378447435 (59h46m10.101010465s) [seg_pos={0 0}, upload=57334]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:22:00.661793613 (59h46m10.384356653s) [upload=57334]: Done uploading segments... Jul 26 23:22:00.661831605 (59h46m10.384394645s) [upload=57334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.285686891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.250758726s FAIL (2025-07-26 23:21:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== YZCV5E3D4FTDJDQ4AOBOLPZG46A, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426412636 (59h44m59.148975666s) [noise=true , address=104.36.82.56:4238 , piece_id=2L3EKIWVFPAWWPDXS6LSYMLFFWKSOHPG7KCJZWDSGOMPNG7FIN7Q, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426760120 (59h44m59.14932315s) [noise=true , address=77.90.31.16:4748 , piece_id=A6QIUUU6AXSMLL3A4SG42XXM37TSP3TIZ74NFBHVGJTE6ARIMUBQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426828191 (59h44m59.149391231s) [noise=true , address=104.36.82.52:4089 , piece_id=32FYUVUECR7TBXWPTLKCXB5RCP3G3D4HMSSLYWFTVEC2WJOZK3OA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427024156 (59h44m59.149587196s) [noise=true , address=79.127.201.214:4321 , piece_id=OJYW4S5XCZRR6QFLYQPJ77YJQSGFLEINUKB4YSXT4G6QFNQACTPA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427129037 (59h44m59.149692077s) [noise=true , address=77.90.31.11:4572 , piece_id=5BYPH5YFW3TWDNAI2IADIBDT7JV6CWZKNWXYDHZAJFWVQT7HMH2Q, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427297279 (59h44m59.149860309s) [noise=true , address=64.49.13.140:4514 , piece_id=XE3BJHE3W3YIM2E2GRRIKIRIJOTQZLAGQSMRQ2KAICZF37HD3JNA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427527769 (59h44m59.150090809s) [noise=true , address=45.91.116.9:4267 , piece_id=XV3IOMBNZNOIYDA5B34ZU2N6RY6OCJH2LCOBSBWMS6LNWDQ52URA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427752118 (59h44m59.150315158s) [noise=true , address=79.127.219.51:4212 , piece_id=PNIJ732N6HDEIMJFTTXFUHW4RMXZLLXHYAP2L37DE76I7WQB2SLA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427973913 (59h44m59.150536953s) [noise=true , address=79.127.205.235:4688 , piece_id=WHVNI3X3I7N7MUFMMDNXRLZUGIFXFL6XTMTGOFM2SJ2ALDBOLIGA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428233259 (59h44m59.150796309s) [noise=true , address=79.127.219.38:4003 , piece_id=FQZGA42QM7V6Z6FD3ODTN6U6MOHDY35A67SL74EDANY5KL42JHBA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428439674 (59h44m59.151002714s) [noise=true , address=104.36.82.46:4504 , piece_id=RUOISBTRTRKLA7FNPJC4KSCJJLMBGCP7U5VLQGEEZPYPGZV5D4JQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428635308 (59h44m59.151198348s) [noise=true , address=79.127.205.245:4242 , piece_id=R7JDMGKT4Q3EJMC5IVWZHMWXZ2ETKESQIUIKCWBHFF6GRECNIDWQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428820903 (59h44m59.151383943s) [noise=true , address=79.127.205.246:4258 , piece_id=L5RVEHM62RHOJUXJUUJTROZC5N5IRGRV7WEEIX36DX23Q6OGUPWA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.429448174 (59h44m59.152011214s) [noise=true , address=77.90.31.10:4531 , piece_id=CJLMHTTDAGPC2T6JHVSFIDE2QDNUEVEU73TUR6B52HJS3G3HGPKA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.429840203 (59h44m59.152403243s) [noise=true , address=104.36.82.54:4169 , piece_id=RDIWB7MIGGU3YAGGEGYJWQPA7YJQEB4DPT7KKVWHH6AGKZOTGOWA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.430174423 (59h44m59.152737463s) [noise=true , address=79.127.205.234:4614 , piece_id=XMQ3ZN6HQLNZW3HGKKPKQOBVXU2GCM2OSUR3F5J75AE6IRPRAWZA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.430533580 (59h44m59.15309662s) [noise=true , address=77.90.31.13:4629 , piece_id=W3XIJEG3YRZMJDW3L4LJ6DQ7ZE3VKJTJ6TDAAQC4FZE7I7SBWAKQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:50.514758382 (59h45m0.237321422s) [upload=57324]: Next returned nil segment Jul 26 23:20:50.514777229 (59h45m0.237340269s) [upload=57324]: Waiting for error group managing segments... Jul 26 23:20:50.802192692 (59h45m0.524755732s) [noise=true , address=79.127.205.234:4614 , piece_id=XMQ3ZN6HQLNZW3HGKKPKQOBVXU2GCM2OSUR3F5J75AE6IRPRAWZA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:50.840550742 (59h45m0.563113782s) [noise=true , address=79.127.205.246:4258 , piece_id=L5RVEHM62RHOJUXJUUJTROZC5N5IRGRV7WEEIX36DX23Q6OGUPWA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:50.840862509 (59h45m0.563425549s) [noise=true , address=79.127.205.235:4688 , piece_id=WHVNI3X3I7N7MUFMMDNXRLZUGIFXFL6XTMTGOFM2SJ2ALDBOLIGA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.222771477 (59h45m0.945334517s) [noise=true , address=79.127.205.245:4242 , piece_id=R7JDMGKT4Q3EJMC5IVWZHMWXZ2ETKESQIUIKCWBHFF6GRECNIDWQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.407900064 (59h45m1.130463104s) [noise=true , address=79.127.219.38:4003 , piece_id=FQZGA42QM7V6Z6FD3ODTN6U6MOHDY35A67SL74EDANY5KL42JHBA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.434483416 (59h45m1.157046456s) [noise=true , address=77.90.31.11:4572 , piece_id=5BYPH5YFW3TWDNAI2IADIBDT7JV6CWZKNWXYDHZAJFWVQT7HMH2Q, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:54.067570544 (59h45m3.790133584s) [noise=true , address=79.127.219.51:4212 , piece_id=PNIJ732N6HDEIMJFTTXFUHW4RMXZLLXHYAP2L37DE76I7WQB2SLA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:54.849175391 (59h45m4.571738431s) [noise=true , address=79.127.201.214:4321 , piece_id=OJYW4S5XCZRR6QFLYQPJ77YJQSGFLEINUKB4YSXT4G6QFNQACTPA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:56.418457887 (59h45m6.141020927s) [noise=true , address=45.91.116.9:4267 , piece_id=XV3IOMBNZNOIYDA5B34ZU2N6RY6OCJH2LCOBSBWMS6LNWDQ52URA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:21:00.082920028 (59h45m9.805483079s) [noise=true , address=64.49.13.140:4514 , piece_id=XE3BJHE3W3YIM2E2GRRIKIRIJOTQZLAGQSMRQ2KAICZF37HD3JNA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:21:48.337414289 (59h45m58.059977329s) [noise=true , address=77.90.31.13:4629 , piece_id=W3XIJEG3YRZMJDW3L4LJ6DQ7ZE3VKJTJ6TDAAQC4FZE7I7SBWAKQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:21:48.337692752 (59h45m58.060255792s) [noise=true , address=77.90.31.10:4531 , piece_id=CJLMHTTDAGPC2T6JHVSFIDE2QDNUEVEU73TUR6B52HJS3G3HGPKA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: rpc: dial tcp 77.90.31.10:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337723180 (59h45m58.060286221s) [noise=true , address=104.36.82.46:4504 , piece_id=RUOISBTRTRKLA7FNPJC4KSCJJLMBGCP7U5VLQGEEZPYPGZV5D4JQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:47886->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337747237 (59h45m58.060310277s) [noise=true , address=104.36.82.47:4529 , piece_id=SNZOK5DIYDWTXV3NU2KZRZQ6MYZCV5E3D4FTDJDQ4AOBOLPZG46A, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: write tcp 188.166.225.170:50350->104.36.82.47:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337781702 (59h45m58.060344742s) [noise=true , address=104.36.82.52:4089 , piece_id=32FYUVUECR7TBXWPTLKCXB5RCP3G3D4HMSSLYWFTVEC2WJOZK3OA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 188.166.225.170:53460->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337835235 (59h45m58.060398275s) [noise=true , address=104.36.82.56:4238 , piece_id=2L3EKIWVFPAWWPDXS6LSYMLFFWKSOHPG7KCJZWDSGOMPNG7FIN7Q, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 188.166.225.170:42094->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337844202 (59h45m58.060407242s) [noise=true , address=104.36.82.54:4169 , piece_id=RDIWB7MIGGU3YAGGEGYJWQPA7YJQEB4DPT7KKVWHH6AGKZOTGOWA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337894598 (59h45m58.060457628s) [noise=true , address=77.90.31.16:4748 , piece_id=A6QIUUU6AXSMLL3A4SG42XXM37TSP3TIZ74NFBHVGJTE6ARIMUBQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: write tcp 188.166.225.170:48038->77.90.31.16:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337915729 (59h45m58.060478759s) [seg_pos={0 0}, upload=57324]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:21:48.586441184 (59h45m58.309004214s) [upload=57324]: Done uploading segments... Jul 26 23:21:48.586468096 (59h45m58.309031136s) [upload=57324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.250782823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........X..XX...X.X.X.X.......XX..XX.XX..XXX............XXX] duration: 1m0.001107154s FAIL (2025-07-26 23:21:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001118706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................................XXX.X..X.....XXXXX.] duration: 5.980909552s FAIL (2025-07-26 23:21:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980922528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X........X.X...........................XXXXXXXXXXXXXXXXX] duration: 30.235871814s FAIL (2025-07-26 23:21:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235893866s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.812039486s FAIL (2025-07-26 23:21:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.812064172s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)