Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X..........X.X........X.X.....................] duration: 5.773608517s FAIL (2025-07-26 23:01:14 UTC) orbiter warning: uplink: too many requests Duration: 5.773640779s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXX] duration: 1m0.231613814s FAIL (2025-07-26 23:01:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:04.931282605 (59h24m14.653845655s) [seg_pos={0 0}, upload=57001]: Begin upload segment... Jul 26 23:00:04.931350183 (59h24m14.653913234s) [seg_pos={0 0}, upload=57001]: Done begin upload segment. Jul 26 23:00:04.931384459 (59h24m14.653947509s) [noise=true , address=79.127.219.38:4005 , piece_id=VJ3QRGLNF263VKTMBZADKGP7K5EJWXPZUVLEZF6WI6ABOY7UJS6A, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.931410639 (59h24m14.653973689s) [noise=true , address=45.91.116.11:4121 , piece_id=H3BI4Q2IBHKJAKK2Z2ELBV3MAAN7DWXBYTEQ67VXS5FLIFUPHL5Q, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.931891770 (59h24m14.65445482s) [noise=true , address=45.91.116.14:4222 , piece_id=HEXYJUKVX6UWXWKLZ5TQDWK6QDISEUW6R3XYM7LYKBYFMEVBEZ5A, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.931973576 (59h24m14.654536627s) [noise=true , address=45.91.116.9:4288 , piece_id=JJVEFWQ4UHMHUYBDKYIHK5LCUFKA3LFOSIZO7WQ5QCSP5F5LIRIA, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.932330188 (59h24m14.654893238s) [noise=true , address=77.90.31.10:4545 , piece_id=27MAS3HUCCF3BAZHCWU4KACI2WUIULM5K7S5DX2FM2L3FWWPOQDA, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.932550369 (59h24m14.655113419s) [noise=true , address=77.90.31.13:4644 , piece_id=M2CK4GRPUJONTHHY7PCG2BHJEAR3LUHKYWK3YUQTPU4FAEIL2TIQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.932725665 (59h24m14.655288715s) [noise=true , address=79.127.219.47:4048 , piece_id=VUIPWOBVVTHTBCZW4463AOSWSDGU7D67DZAUKWZNA3GLLHM4UMMQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.933025108 (59h24m14.655588148s) [noise=true , address=79.127.219.51:4212 , piece_id=LVPAU7KECWBCA6OCWVE6VBVPGZUVBSOB3VFA5QLG6W6H4LEGSXMQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.933097526 (59h24m14.655660576s) [noise=true , address=64.49.13.140:4514 , piece_id=UFH7ER5K7MRU5PUTM3HEXXPW6VTPE32646CY3P4JWRE5MZ7YIN2A, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.933578968 (59h24m14.656142008s) [noise=true , address=45.91.116.9:4277 , piece_id=RBQQ4SEFADXHHSXHQERQADWT3H6OWJLJBYJ4HZQKZJ6LPPIACZGQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.933614736 (59h24m14.656177786s) [noise=true , address=79.127.205.234:4610 , piece_id=7PX4NVR2TCPPKLWNRRFENTPF5YTFVWJNFPV46LQCR2TQJYFQN57A, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934010573 (59h24m14.656573613s) [noise=true , address=77.90.31.16:4743 , piece_id=DIE3OKAOAXACX3RQ6AQHMC3IJL4YPDVUHREBJIEPWJHUVZYBIDWQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934044307 (59h24m14.656607347s) [noise=true , address=79.127.205.245:4249 , piece_id=QYCV6NH25XOO5OKNONYLAGG7E3RABVCEV4O2MAMC26ZHKKD3ZSUQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934337649 (59h24m14.656900689s) [noise=true , address=79.127.205.245:4224 , piece_id=XROXS5R7LOSRJ3T3WLXMTU35JSRNNYCJWZ7XR56MIQLUCAAPO5YQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934542310 (59h24m14.65710536s) [noise=true , address=45.91.116.14:4206 , piece_id=M52PCBKHCX6VZAGKSC2Q5P2YPROQBXOGNIS4YESIOLNGSBPKGIOQ, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934650737 (59h24m14.657213787s) [noise=true , address=45.91.116.11:4124 , piece_id=L2VKZ45EC6I73J3DBMGUIZP5QKTEH2DKWCAA3WXE34FCLUGMODFA, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.934986440 (59h24m14.657549491s) [noise=true , address=79.127.205.235:4670 , piece_id=BXL73F5PHF7TSKIP5CH6ZRP7JAQOAHRUQW62MBZDAR5EIN3AWIFA, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:04.935040594 (59h24m14.657603644s) [noise=true , address=79.127.219.38:4015 , piece_id=JZIF436PSHCLHLWBNTOXLPJGZRZZBJ3SB445LE6V3F4DMZZ3755Q, seg_pos={0 0}, upload=57001]: Uploading piece... Jul 26 23:00:06.013261781 (59h24m15.735824831s) [noise=true , address=79.127.219.51:4212 , piece_id=LVPAU7KECWBCA6OCWVE6VBVPGZUVBSOB3VFA5QLG6W6H4LEGSXMQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:06.040824761 (59h24m15.763387811s) [upload=57001]: Next returned nil segment Jul 26 23:00:06.040847916 (59h24m15.763410966s) [upload=57001]: Waiting for error group managing segments... Jul 26 23:00:06.326922629 (59h24m16.049485679s) [noise=true , address=79.127.205.245:4224 , piece_id=XROXS5R7LOSRJ3T3WLXMTU35JSRNNYCJWZ7XR56MIQLUCAAPO5YQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:06.333836475 (59h24m16.056399525s) [noise=true , address=79.127.205.234:4610 , piece_id=7PX4NVR2TCPPKLWNRRFENTPF5YTFVWJNFPV46LQCR2TQJYFQN57A, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:06.360393993 (59h24m16.082957033s) [noise=true , address=79.127.205.235:4670 , piece_id=BXL73F5PHF7TSKIP5CH6ZRP7JAQOAHRUQW62MBZDAR5EIN3AWIFA, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:06.496545276 (59h24m16.219108316s) [noise=true , address=79.127.205.245:4249 , piece_id=QYCV6NH25XOO5OKNONYLAGG7E3RABVCEV4O2MAMC26ZHKKD3ZSUQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:06.935504132 (59h24m16.658067173s) [noise=true , address=45.91.116.14:4206 , piece_id=M52PCBKHCX6VZAGKSC2Q5P2YPROQBXOGNIS4YESIOLNGSBPKGIOQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:07.002589384 (59h24m16.725152434s) [noise=true , address=45.91.116.9:4288 , piece_id=JJVEFWQ4UHMHUYBDKYIHK5LCUFKA3LFOSIZO7WQ5QCSP5F5LIRIA, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:07.409571986 (59h24m17.132135037s) [noise=true , address=45.91.116.9:4277 , piece_id=RBQQ4SEFADXHHSXHQERQADWT3H6OWJLJBYJ4HZQKZJ6LPPIACZGQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:07.553153424 (59h24m17.275716464s) [noise=true , address=45.91.116.11:4124 , piece_id=L2VKZ45EC6I73J3DBMGUIZP5QKTEH2DKWCAA3WXE34FCLUGMODFA, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr, address:45.91.116.11:4124): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:08.279091344 (59h24m18.001654384s) [noise=true , address=79.127.219.38:4015 , piece_id=JZIF436PSHCLHLWBNTOXLPJGZRZZBJ3SB445LE6V3F4DMZZ3755Q, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:08.488289003 (59h24m18.210852043s) [noise=true , address=45.91.116.14:4222 , piece_id=HEXYJUKVX6UWXWKLZ5TQDWK6QDISEUW6R3XYM7LYKBYFMEVBEZ5A, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:08.583804273 (59h24m18.306367313s) [noise=true , address=64.49.13.140:4514 , piece_id=UFH7ER5K7MRU5PUTM3HEXXPW6VTPE32646CY3P4JWRE5MZ7YIN2A, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:09.814489040 (59h24m19.53705208s) [noise=true , address=45.91.116.11:4121 , piece_id=H3BI4Q2IBHKJAKK2Z2ELBV3MAAN7DWXBYTEQ67VXS5FLIFUPHL5Q, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:10.015537149 (59h24m19.738100189s) [noise=true , address=79.127.219.47:4048 , piece_id=VUIPWOBVVTHTBCZW4463AOSWSDGU7D67DZAUKWZNA3GLLHM4UMMQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:00:14.958044085 (59h24m24.680607135s) [noise=true , address=79.127.219.38:4005 , piece_id=VJ3QRGLNF263VKTMBZADKGP7K5EJWXPZUVLEZF6WI6ABOY7UJS6A, seg_pos={0 0}, upload=57001]: Done uploading piece. err: <nil> Jul 26 23:01:03.843815861 (59h25m13.566378901s) [noise=true , address=77.90.31.10:4545 , piece_id=27MAS3HUCCF3BAZHCWU4KACI2WUIULM5K7S5DX2FM2L3FWWPOQDA, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: write tcp 188.166.225.170:45392->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:03.843856138 (59h25m13.566419178s) [noise=true , address=77.90.31.13:4644 , piece_id=M2CK4GRPUJONTHHY7PCG2BHJEAR3LUHKYWK3YUQTPU4FAEIL2TIQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 188.166.225.170:43642->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:03.843905563 (59h25m13.566468603s) [noise=true , address=77.90.31.16:4743 , piece_id=DIE3OKAOAXACX3RQ6AQHMC3IJL4YPDVUHREBJIEPWJHUVZYBIDWQ, seg_pos={0 0}, upload=57001]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:03.843929749 (59h25m13.566492789s) [seg_pos={0 0}, upload=57001]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:01:04.074171617 (59h25m13.796734657s) [upload=57001]: Done uploading segments... Jul 26 23:01:04.074198158 (59h25m13.796761198s) [upload=57001]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 1m0.231626761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...........X...X...............X..XX...X.X.X.X.......XX..XX.] duration: 19.043891069s FAIL (2025-07-26 23:00:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.043904114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.........X...........] duration: 5.745457769s FAIL (2025-07-26 23:00:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.745497226s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............X...X...............X..XX...X.X.X.X.......XX..XX] duration: 1m0.001130103s FAIL (2025-07-26 23:00:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001139382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X..........X.X........X.X....................] duration: 5.699240169s FAIL (2025-07-26 23:00:18 UTC) orbiter warning: uplink: too many requests Duration: 5.699267661s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXX] duration: 1m0.232346894s FAIL (2025-07-26 23:00:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=56987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:59:04.479777446 (59h23m14.202340486s) [seg_pos={0 0}, upload=56987]: Begin upload segment... Jul 26 22:59:04.479817052 (59h23m14.202380092s) [seg_pos={0 0}, upload=56987]: Done begin upload segment. Jul 26 22:59:04.479834926 (59h23m14.202397966s) [noise=true , address=79.127.219.38:4018 , piece_id=I75SV4TWYFCWWPZ57WFXSEIY6GPJU3O4R4PQGYNNZ46HIUVXEG6A, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.479912083 (59h23m14.202475124s) [noise=true , address=77.90.31.13:4651 , piece_id=DK5VU2ZCLHRPDQCU7I4QRFT34MG4UZZTFBOXTCALOEX5SRKZEI5A, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.480217067 (59h23m14.202780107s) [noise=true , address=77.90.31.21:4902 , piece_id=NX2NMSZ63FRRPCWOBPBJAJXK42CGFBEB7AN6N3AP3NZS2J3K4Q3A, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.480477666 (59h23m14.203040706s) [noise=true , address=77.90.31.16:4767 , piece_id=HMPUGTBIQJTYZLDCMZH5B3RKHHASW6OK4F2COG4HU2SZZWKMMRZQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.480539674 (59h23m14.203102714s) [noise=true , address=79.127.219.47:4048 , piece_id=3PQPSD6PC4KC3SIFLJPAT6JSG3HIXP4MCOZKNU7HYWTITL5PKKHQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.480807386 (59h23m14.203370426s) [noise=true , address=45.91.116.14:4206 , piece_id=DT5B6ISWYJEDC47CQY3STRQAZ3YQE5RVDPQFYQRWKTXPEPOEZTIA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.480839188 (59h23m14.203402228s) [noise=true , address=79.127.205.235:4679 , piece_id=33UZVJ3CFUE7ZYWV32NGFJDNU4SQVLMBZDQLAGVGTUYL2BYLIRRQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.481079577 (59h23m14.203642617s) [noise=true , address=77.90.31.15:4714 , piece_id=WERL5WGBFDPVT3PG3F2GUX7AJAF2J74IFLO3PFIWT7DNDKRX7I6Q, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.481197934 (59h23m14.203760974s) [noise=true , address=79.127.205.234:4606 , piece_id=6T5PXXGM6QXTNZCS54H4TXKKXHFFQZJCD6KB3V2MOFWMESGRS5PQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.481378629 (59h23m14.203941669s) [noise=true , address=64.49.13.140:4514 , piece_id=E4HCFQDUHMS2YANLH7FGS4QUPLAW35RNBHRONDMCQ77XPNFHZLLA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.481714191 (59h23m14.204277221s) [noise=true , address=79.127.205.245:4247 , piece_id=WYKUHKR67QOZUCF4RJAY3CVXSO2BLGJILM5QBGVVHPPNPOYSULGA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.481739069 (59h23m14.204302099s) [noise=true , address=79.127.205.233:4633 , piece_id=2KA3MZVFYMYYQR7THLUKMF5OIMUBESRYGWLOS7NDB4FNZ33YVA5A, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.482012111 (59h23m14.204575151s) [noise=true , address=77.90.31.13:4644 , piece_id=N7MOLLWJAR3VH3N3JYAEUU7O2OZAPCAY3LJOMML6TUGCLO67UYPQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.482030767 (59h23m14.204593807s) [noise=true , address=77.90.31.11:4574 , piece_id=MSKU2LB2F4KD222MCDOXJMJA2IKLYFSE2EMLVIMGQOJCMW4KN2CA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.482392559 (59h23m14.204955599s) [noise=true , address=79.127.219.38:4007 , piece_id=MRZMOX6XO7EPAE5F6NIBNP6L42UJ32GV4JT6H3CDHFPBLU333AAA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.482670641 (59h23m14.205233681s) [noise=true , address=79.127.205.245:4225 , piece_id=3BECSXXN3XMGMQDGMA2LXKK5IACKVMVLUG34JNW5H5MYQMOT22WA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.484060009 (59h23m14.206623049s) [noise=true , address=77.90.31.19:4875 , piece_id=JKK36BQVILKVNZJNU3QSG3DURGJ4TJXT37HE3KAGNNSFNSFCEMJA, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:04.484888093 (59h23m14.207451133s) [noise=true , address=77.90.31.16:4736 , piece_id=LSAJDCTYDCL5YY646UDZQSU6JS4PHQG35IZPQ2CWZLV2ZTSZNDUQ, seg_pos={0 0}, upload=56987]: Uploading piece... Jul 26 22:59:05.586399276 (59h23m15.308962316s) [upload=56987]: Next returned nil segment Jul 26 22:59:05.586418053 (59h23m15.308981093s) [upload=56987]: Waiting for error group managing segments... Jul 26 22:59:05.722572832 (59h23m15.445135872s) [noise=true , address=64.49.13.140:4514 , piece_id=E4HCFQDUHMS2YANLH7FGS4QUPLAW35RNBHRONDMCQ77XPNFHZLLA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:05.871935940 (59h23m15.59449898s) [noise=true , address=79.127.205.245:4225 , piece_id=3BECSXXN3XMGMQDGMA2LXKK5IACKVMVLUG34JNW5H5MYQMOT22WA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:05.909542646 (59h23m15.632105686s) [noise=true , address=79.127.205.234:4606 , piece_id=6T5PXXGM6QXTNZCS54H4TXKKXHFFQZJCD6KB3V2MOFWMESGRS5PQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:05.911694834 (59h23m15.634257894s) [noise=true , address=79.127.205.233:4633 , piece_id=2KA3MZVFYMYYQR7THLUKMF5OIMUBESRYGWLOS7NDB4FNZ33YVA5A, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:05.924019853 (59h23m15.646582893s) [noise=true , address=79.127.205.235:4679 , piece_id=33UZVJ3CFUE7ZYWV32NGFJDNU4SQVLMBZDQLAGVGTUYL2BYLIRRQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:06.066422876 (59h23m15.788985916s) [noise=true , address=79.127.205.245:4247 , piece_id=WYKUHKR67QOZUCF4RJAY3CVXSO2BLGJILM5QBGVVHPPNPOYSULGA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:06.409276282 (59h23m16.131839322s) [noise=true , address=79.127.219.38:4007 , piece_id=MRZMOX6XO7EPAE5F6NIBNP6L42UJ32GV4JT6H3CDHFPBLU333AAA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:06.561979140 (59h23m16.28454218s) [noise=true , address=77.90.31.21:4902 , piece_id=NX2NMSZ63FRRPCWOBPBJAJXK42CGFBEB7AN6N3AP3NZS2J3K4Q3A, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:08.710333822 (59h23m18.432896862s) [noise=true , address=79.127.219.38:4018 , piece_id=I75SV4TWYFCWWPZ57WFXSEIY6GPJU3O4R4PQGYNNZ46HIUVXEG6A, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:08.711411162 (59h23m18.433974202s) [noise=true , address=79.127.219.47:4048 , piece_id=3PQPSD6PC4KC3SIFLJPAT6JSG3HIXP4MCOZKNU7HYWTITL5PKKHQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:17.840447937 (59h23m27.563010977s) [noise=true , address=45.91.116.14:4206 , piece_id=DT5B6ISWYJEDC47CQY3STRQAZ3YQE5RVDPQFYQRWKTXPEPOEZTIA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 22:59:34.344430176 (59h23m44.066993217s) [noise=true , address=77.90.31.19:4875 , piece_id=JKK36BQVILKVNZJNU3QSG3DURGJ4TJXT37HE3KAGNNSFNSFCEMJA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: <nil> Jul 26 23:00:03.372154267 (59h24m13.094717317s) [noise=true , address=77.90.31.11:4574 , piece_id=MSKU2LB2F4KD222MCDOXJMJA2IKLYFSE2EMLVIMGQOJCMW4KN2CA, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:00:03.372457026 (59h24m13.095020077s) [noise=true , address=77.90.31.16:4767 , piece_id=HMPUGTBIQJTYZLDCMZH5B3RKHHASW6OK4F2COG4HU2SZZWKMMRZQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:03.372513895 (59h24m13.095076945s) [noise=true , address=77.90.31.15:4714 , piece_id=WERL5WGBFDPVT3PG3F2GUX7AJAF2J74IFLO3PFIWT7DNDKRX7I6Q, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: write tcp 188.166.225.170:60210->77.90.31.15:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:03.372675224 (59h24m13.095238264s) [noise=true , address=77.90.31.16:4736 , piece_id=LSAJDCTYDCL5YY646UDZQSU6JS4PHQG35IZPQ2CWZLV2ZTSZNDUQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:03.372685473 (59h24m13.095248514s) [noise=true , address=77.90.31.13:4644 , piece_id=N7MOLLWJAR3VH3N3JYAEUU7O2OZAPCAY3LJOMML6TUGCLO67UYPQ, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 188.166.225.170:37750->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:03.372725500 (59h24m13.09528854s) [noise=true , address=77.90.31.13:4651 , piece_id=DK5VU2ZCLHRPDQCU7I4QRFT34MG4UZZTFBOXTCALOEX5SRKZEI5A, seg_pos={0 0}, upload=56987]: Done uploading piece. err: ecclient: upload failed (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, address:77.90.31.13:4651): protocol: rpc: dial tcp 77.90.31.13:4651: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:03.372754576 (59h24m13.095317616s) [seg_pos={0 0}, upload=56987]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:00:03.602136780 (59h24m13.32469983s) [upload=56987]: Done uploading segments... Jul 26 23:00:03.602180984 (59h24m13.324744044s) [upload=56987]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.232367252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.........X..........] duration: 7.019712544s FAIL (2025-07-26 22:59:43 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 7.01973698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.............X...X...............X..XX...X.X.X.X.......XX..X] duration: 1m0.001021212s FAIL (2025-07-26 22:59:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001037844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X..........X.X........X.X...................] duration: 5.76721568s FAIL (2025-07-26 22:59:24 UTC) orbiter warning: uplink: too many requests Duration: 5.76724236s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.........X.........] duration: 5.833422829s FAIL (2025-07-26 22:59:09 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.833454598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXX] duration: 1m0.232540946s FAIL (2025-07-26 22:59:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 59h22m13.294097948s) [upload=56969]: Starting upload Jul 26 22:58:03.571586958 (59h22m13.294149998s) [upload=56969]: Done starting upload Jul 26 22:58:03.571741133 (59h22m13.294304173s) [upload=56969]: Uploading segments... Jul 26 22:58:03.571992875 (59h22m13.294555945s) [upload=56969]: Got next segment. Inline: false Jul 26 22:58:03.572098497 (59h22m13.294661577s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:58:03.572106422 (59h22m13.294669472s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:58:04.083650010 (59h22m13.80621304s) [seg_pos={0 0}, upload=56969]: Begin upload segment... Jul 26 22:58:04.083701128 (59h22m13.806264158s) [seg_pos={0 0}, upload=56969]: Done begin upload segment. Jul 26 22:58:04.083725996 (59h22m13.806289026s) [noise=true , address=77.90.31.10:4546 , piece_id=UOMP6SA2QVXNRDQV3JXJGORUSNE2VB3ERJ2ZE4C34ODN356XHD7A, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.083746375 (59h22m13.806309425s) [noise=true , address=79.127.205.246:4280 , piece_id=OMSUMCU66R73WCLJTGSIV3RHC6HMYTRLW3NYXKBP7IFJ3HZNS7BQ, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.084138265 (59h22m13.806701305s) [noise=true , address=77.90.31.21:4891 , piece_id=G4HKUUWK7WMRHNXEQETAO3J7TOKAPV2CHZGVVGIIBLPHHWOHQLGQ, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.084149786 (59h22m13.806712826s) [noise=true , address=79.127.219.51:4212 , piece_id=OARJMFUVEXV2PWKUDGWBMICI37NCYVH3K7KW46LCNATHCMTQYIHQ, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.084575151 (59h22m13.807138201s) [noise=true , address=64.49.13.170:4718 , piece_id=LFFAPDOXOOZ4NFLQDCNXQF2T4RFS65FXXRZSSSPAVOTGHU5DH6EA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.084874383 (59h22m13.807437423s) [noise=true , address=77.90.31.16:4736 , piece_id=3RS6XJROLWWKS2PLEZLGSUSSWT43RP6DVGRXCWYKFGFDKQKWPJEA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085120584 (59h22m13.807683624s) [noise=true , address=64.49.13.138:4483 , piece_id=DG5RQNSL6AYWC2D463TX5Y6CH7JERG7ZELSLZBAGHOPQ4Y7XSLXA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085437620 (59h22m13.80800065s) [noise=true , address=79.127.219.38:4007 , piece_id=55I4WYLHTJ3AFIWJN5UXTG3FQELIROBZFDW6THR5EQ5MZ7OI6ONA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085614679 (59h22m13.808177719s) [noise=true , address=77.90.31.13:4651 , piece_id=65P47SU6H3QLNPMY5J2EDFXNPBDCQKQ3OAXAFCTQFX566V5NQT2A, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085701215 (59h22m13.808264255s) [noise=true , address=77.90.31.11:4569 , piece_id=NKAZCFRWSSWBTLGA6UOIMDSTLOQVMLLH7OS72DO7JFBDYRPY2DUA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085880658 (59h22m13.808443698s) [noise=true , address=79.127.201.215:4341 , piece_id=YWVELYAQO4FEIFNDHLJSMS7SLLEFIZEPJ6JLSNPPJUZFOXFN4A4Q, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.085948047 (59h22m13.808511087s) [noise=true , address=45.91.116.9:4277 , piece_id=TU7V2MSNJNWL25ZFEFYWEHZWYACKLGFDSHPFQSOX6KIUBU5SZ67A, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.086094537 (59h22m13.808657567s) [noise=true , address=79.127.205.234:4612 , piece_id=EG6A2CL325MBBNJNFNWARARNV6OV76UB7SZNFOSB2I3DEVVB3LKA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.086299429 (59h22m13.808862469s) [noise=true , address=156.146.43.225:4449 , piece_id=HQ4K55E7VUXTQ6VAJZ24T6XOD25XUULRP3RR6MRVXOBHCDHRTLVQ, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.086478983 (59h22m13.809042023s) [noise=true , address=77.90.31.13:4658 , piece_id=L7TFX7HPFEYVI6XAPNQ3O6YGHI7WTRE2FLX5R647DSNNLPHAVW2A, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.086629731 (59h22m13.809192771s) [noise=true , address=45.91.116.14:4194 , piece_id=K6WIFEJTD6CAITANCQCU4KV7I7WIZCK57AZB7S2X6CRTDJSFSRCA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.086843670 (59h22m13.809406711s) [noise=true , address=79.127.219.47:4061 , piece_id=6NEFVTGDZZLRBU5PRZ4P7VOSMRQNZRJM56BF7E52GFIG3DJTLKGA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:04.087058111 (59h22m13.809621151s) [noise=true , address=77.90.31.21:4887 , piece_id=3QZFJ24CW2E4P6FDS5FOZY6OWT2VW54LSY77NQBGFWIKU4F7GATA, seg_pos={0 0}, upload=56969]: Uploading piece... Jul 26 22:58:05.117894836 (59h22m14.840457876s) [noise=true , address=79.127.201.215:4341 , piece_id=YWVELYAQO4FEIFNDHLJSMS7SLLEFIZEPJ6JLSNPPJUZFOXFN4A4Q, seg_pos={0 0}, upload=56969]: Done uploading piece. err: ecclient: upload failed (node:12Pvou3YLLA7MA8WAfqjWLRwgwDXWPWZKGGRWYAVBkKCxF5mciG, address:79.127.201.215:4341): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:05.209263274 (59h22m14.931826314s) [upload=56969]: Next returned nil segment Jul 26 22:58:05.209279937 (59h22m14.931842977s) [upload=56969]: Waiting for error group managing segments... Jul 26 22:58:05.420824004 (59h22m15.143387044s) [noise=true , address=79.127.219.47:4061 , piece_id=6NEFVTGDZZLRBU5PRZ4P7VOSMRQNZRJM56BF7E52GFIG3DJTLKGA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:05.496952246 (59h22m15.219515286s) [noise=true , address=79.127.205.234:4612 , piece_id=EG6A2CL325MBBNJNFNWARARNV6OV76UB7SZNFOSB2I3DEVVB3LKA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:05.538051583 (59h22m15.260614623s) [noise=true , address=79.127.205.246:4280 , piece_id=OMSUMCU66R73WCLJTGSIV3RHC6HMYTRLW3NYXKBP7IFJ3HZNS7BQ, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:05.952146583 (59h22m15.674709623s) [noise=true , address=79.127.219.38:4007 , piece_id=55I4WYLHTJ3AFIWJN5UXTG3FQELIROBZFDW6THR5EQ5MZ7OI6ONA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:06.020839390 (59h22m15.74340244s) [noise=true , address=77.90.31.21:4891 , piece_id=G4HKUUWK7WMRHNXEQETAO3J7TOKAPV2CHZGVVGIIBLPHHWOHQLGQ, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:06.124326192 (59h22m15.846889232s) [noise=true , address=77.90.31.11:4569 , piece_id=NKAZCFRWSSWBTLGA6UOIMDSTLOQVMLLH7OS72DO7JFBDYRPY2DUA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:06.143550159 (59h22m15.866113189s) [noise=true , address=77.90.31.21:4887 , piece_id=3QZFJ24CW2E4P6FDS5FOZY6OWT2VW54LSY77NQBGFWIKU4F7GATA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:06.295315964 (59h22m16.017878994s) [noise=true , address=79.127.219.51:4212 , piece_id=OARJMFUVEXV2PWKUDGWBMICI37NCYVH3K7KW46LCNATHCMTQYIHQ, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:08.792875574 (59h22m18.515438614s) [noise=true , address=156.146.43.225:4449 , piece_id=HQ4K55E7VUXTQ6VAJZ24T6XOD25XUULRP3RR6MRVXOBHCDHRTLVQ, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:09.256820611 (59h22m18.979383651s) [noise=true , address=45.91.116.9:4277 , piece_id=TU7V2MSNJNWL25ZFEFYWEHZWYACKLGFDSHPFQSOX6KIUBU5SZ67A, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:11.712769531 (59h22m21.435332571s) [noise=true , address=64.49.13.138:4483 , piece_id=DG5RQNSL6AYWC2D463TX5Y6CH7JERG7ZELSLZBAGHOPQ4Y7XSLXA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:12.413902537 (59h22m22.136465577s) [noise=true , address=64.49.13.170:4718 , piece_id=LFFAPDOXOOZ4NFLQDCNXQF2T4RFS65FXXRZSSSPAVOTGHU5DH6EA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:21.325450958 (59h22m31.048013998s) [noise=true , address=45.91.116.14:4194 , piece_id=K6WIFEJTD6CAITANCQCU4KV7I7WIZCK57AZB7S2X6CRTDJSFSRCA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:58:56.545140787 (59h23m6.267703827s) [noise=true , address=77.90.31.10:4546 , piece_id=UOMP6SA2QVXNRDQV3JXJGORUSNE2VB3ERJ2ZE4C34ODN356XHD7A, seg_pos={0 0}, upload=56969]: Done uploading piece. err: <nil> Jul 26 22:59:02.919754688 (59h23m12.642317718s) [noise=true , address=77.90.31.13:4651 , piece_id=65P47SU6H3QLNPMY5J2EDFXNPBDCQKQ3OAXAFCTQFX566V5NQT2A, seg_pos={0 0}, upload=56969]: Done uploading piece. err: ecclient: upload failed (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, address:77.90.31.13:4651): protocol: write tcp 188.166.225.170:51300->77.90.31.13:4651: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:02.919844140 (59h23m12.64240717s) [noise=true , address=77.90.31.16:4736 , piece_id=3RS6XJROLWWKS2PLEZLGSUSSWT43RP6DVGRXCWYKFGFDKQKWPJEA, seg_pos={0 0}, upload=56969]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; context deadline exceeded Jul 26 22:59:02.919898744 (59h23m12.642461784s) [noise=true , address=77.90.31.13:4658 , piece_id=L7TFX7HPFEYVI6XAPNQ3O6YGHI7WTRE2FLX5R647DSNNLPHAVW2A, seg_pos={0 0}, upload=56969]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 188.166.225.170:42602->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:02.919976762 (59h23m12.642539793s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:59:03.150316298 (59h23m12.872879338s) [upload=56969]: Done uploading segments... Jul 26 22:59:03.150339613 (59h23m12.872902653s) [upload=56969]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.232561916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X..........X.X........X.X..................] duration: 5.741716352s FAIL (2025-07-26 22:58:34 UTC) orbiter warning: uplink: too many requests Duration: 5.741743766s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..............X...X...............X..XX...X.X.X.X.......XX..] duration: 8.55965357s FAIL (2025-07-26 22:58:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.559680151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...............X...X...............X..XX...X.X.X.X.......XX.] duration: 42.68064837s FAIL (2025-07-26 22:58:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.680668799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXX] duration: 1m0.23595647s FAIL (2025-07-26 22:58:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94137s) [upload=56936]: Done uploading segments... Jul 26 22:57:02.022399651 (59h21m11.744962691s) [upload=56936]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 22:57:02.910786693 (59h21m12.633349733s) [upload=56951]: Starting upload Jul 26 22:57:02.910859433 (59h21m12.633422473s) [upload=56951]: Done starting upload Jul 26 22:57:02.910942652 (59h21m12.633505692s) [upload=56951]: Uploading segments... Jul 26 22:57:02.911088250 (59h21m12.63365129s) [upload=56951]: Got next segment. Inline: false Jul 26 22:57:02.911131313 (59h21m12.633694373s) [upload=56951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:57:02.911137435 (59h21m12.633700475s) [upload=56951]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:57:03.366315841 (59h21m13.088878871s) [seg_pos={0 0}, upload=56951]: Begin upload segment... Jul 26 22:57:03.366366979 (59h21m13.08893008s) [seg_pos={0 0}, upload=56951]: Done begin upload segment. Jul 26 22:57:03.366399071 (59h21m13.088962111s) [noise=true , address=79.127.201.215:4357 , piece_id=JPAUU4HTOYMQMFY3ODN5YYYLNSOIBZ2IFIY6UTHFR6XDFFO6FWOQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.366496518 (59h21m13.089059558s) [noise=true , address=79.127.219.38:4005 , piece_id=A4ZIZV63ZPBCSKVC4YZ3S7Z72GQZZWNVXQNOJVJL4ZYFP5ODA6HQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.366817160 (59h21m13.0893802s) [noise=true , address=77.90.31.15:4722 , piece_id=N3AHIRSEAGBNUQHURAJRH5HYHRC4H4QAGFPLDPKUM2WU65XUVTHA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367088399 (59h21m13.08965144s) [noise=true , address=77.90.31.10:4548 , piece_id=UZVX3Z7UPKDUCAAGFHHVGXLA4FPLEKYISWE7PIC5PJDEA5MY5EEQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367150077 (59h21m13.089713117s) [noise=true , address=64.49.13.169:4716 , piece_id=TXRBUZCBYQBCIRW2GS527MG7U6KKKKF3MVPAJ6VY2XR6777IAIOA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367353407 (59h21m13.089916447s) [noise=true , address=64.49.13.170:4719 , piece_id=SYRCXBG52F26QIQNZ6SZV6U2QE6WN2G6VWMS3SVPLPM375UZN3PA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367492052 (59h21m13.090055092s) [noise=true , address=79.127.205.233:4642 , piece_id=TZ4DKU742UWT2PG3JCJJNMQ275H534723MXSQ44J454CCHIWD77Q, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367625478 (59h21m13.090188508s) [noise=true , address=77.90.31.10:4523 , piece_id=4MHJ7S7PVPSLORYVSIQDBMYHDCHOFQHDBDAUEH3B3VL5ZIZNEE2Q, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367798078 (59h21m13.090361118s) [noise=true , address=45.91.116.8:4065 , piece_id=ZFC5NG6WZ5XWVGJ4IAYMX4P3NO3XW722YCQJBPNTRAMKLPKSK3VQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.367842313 (59h21m13.090405353s) [noise=true , address=79.127.205.245:4245 , piece_id=EHHHSZ5HXVVIPJXHQXS3HFC7YNNWBC32BZBFQ5R4HX27QMMWH2HA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368096398 (59h21m13.090659438s) [noise=true , address=79.127.220.101:4048 , piece_id=XLICLHKFI5LQZZUGMARSQH7D63ZD6UVFPZQNT6UPI2CGWQ6HKWKA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368103422 (59h21m13.090666462s) [noise=true , address=79.127.205.245:4226 , piece_id=IVZXOT2HSNWL7RJTTRNAS5LGHABGP4UQ5WWUIYXAN7M2YR3XHAXQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368356766 (59h21m13.090919806s) [noise=true , address=156.146.43.225:4448 , piece_id=RQO7VV2LO3HXX5ZQ5LOWBP7CQZWDZ5KGELG4SJNAKRCQAK3FBY5A, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368528144 (59h21m13.091091174s) [noise=true , address=79.127.201.215:4350 , piece_id=JGTR3XDCZSTWU7CK4TLVZ23M7TKEOSANBKC5BRHW4TBH6C76Q5IQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368590524 (59h21m13.091153564s) [noise=true , address=79.127.220.97:4361 , piece_id=ZEPO7TFYGBAYVJ4HOAVUQYT5FXIHXC67K7WVM7I46U5FPWHQ35TQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368781259 (59h21m13.091344299s) [noise=true , address=79.127.219.51:4212 , piece_id=KPCVEZSQJUHGPW7KYQ5RTG2FMPSPXAISWEQPHZVD7DFPNQALFYNA, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.368813771 (59h21m13.091376811s) [noise=true , address=79.127.205.246:4273 , piece_id=G4HLZ7X32M6BA4TK3RLMS5C4TRTA4JPXQ4DAL4ATHLLCWCRPDQGQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:03.369043971 (59h21m13.091607001s) [noise=true , address=79.127.205.234:4614 , piece_id=YBG4X2TIJAAJLCFLGW64ELP5SKL7IFPMZJGCWK7BEMHQSASY5IUQ, seg_pos={0 0}, upload=56951]: Uploading piece... Jul 26 22:57:04.377489569 (59h21m14.10005261s) [noise=true , address=79.127.201.215:4350 , piece_id=JGTR3XDCZSTWU7CK4TLVZ23M7TKEOSANBKC5BRHW4TBH6C76Q5IQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: ecclient: upload failed (node:1WhZAWvHCTj5xVkUuVZtgkFB1dhFKpBPq18YMKkepFx5qxkUAr, address:79.127.201.215:4350): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:04.400352288 (59h21m14.122915328s) [noise=true , address=156.146.43.225:4448 , piece_id=RQO7VV2LO3HXX5ZQ5LOWBP7CQZWDZ5KGELG4SJNAKRCQAK3FBY5A, seg_pos={0 0}, upload=56951]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:04.444831172 (59h21m14.167394212s) [noise=true , address=79.127.219.51:4212 , piece_id=KPCVEZSQJUHGPW7KYQ5RTG2FMPSPXAISWEQPHZVD7DFPNQALFYNA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:04.471280913 (59h21m14.193843943s) [upload=56951]: Next returned nil segment Jul 26 22:57:04.471298246 (59h21m14.193861286s) [upload=56951]: Waiting for error group managing segments... Jul 26 22:57:04.573241265 (59h21m14.295804335s) [noise=true , address=79.127.201.215:4357 , piece_id=JPAUU4HTOYMQMFY3ODN5YYYLNSOIBZ2IFIY6UTHFR6XDFFO6FWOQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: ecclient: upload failed (node:1yXKsyPEVG4W52L7SUmsDGt4oD2D9YyqsfwDVWpjMnMnyVo2Da, address:79.127.201.215:4357): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:04.766841954 (59h21m14.489404984s) [noise=true , address=79.127.205.234:4614 , piece_id=YBG4X2TIJAAJLCFLGW64ELP5SKL7IFPMZJGCWK7BEMHQSASY5IUQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:04.774605808 (59h21m14.497168848s) [noise=true , address=79.127.205.245:4245 , piece_id=EHHHSZ5HXVVIPJXHQXS3HFC7YNNWBC32BZBFQ5R4HX27QMMWH2HA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:04.819044644 (59h21m14.541607684s) [noise=true , address=79.127.205.246:4273 , piece_id=G4HLZ7X32M6BA4TK3RLMS5C4TRTA4JPXQ4DAL4ATHLLCWCRPDQGQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:04.884131872 (59h21m14.606694913s) [noise=true , address=79.127.205.245:4226 , piece_id=IVZXOT2HSNWL7RJTTRNAS5LGHABGP4UQ5WWUIYXAN7M2YR3XHAXQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:04.950650871 (59h21m14.673213921s) [noise=true , address=79.127.205.233:4642 , piece_id=TZ4DKU742UWT2PG3JCJJNMQ275H534723MXSQ44J454CCHIWD77Q, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:05.319920285 (59h21m15.042483325s) [noise=true , address=79.127.219.38:4005 , piece_id=A4ZIZV63ZPBCSKVC4YZ3S7Z72GQZZWNVXQNOJVJL4ZYFP5ODA6HQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:07.632695972 (59h21m17.355259013s) [noise=true , address=64.49.13.170:4719 , piece_id=SYRCXBG52F26QIQNZ6SZV6U2QE6WN2G6VWMS3SVPLPM375UZN3PA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:07.711456050 (59h21m17.43401909s) [noise=true , address=64.49.13.169:4716 , piece_id=TXRBUZCBYQBCIRW2GS527MG7U6KKKKF3MVPAJ6VY2XR6777IAIOA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:07.982860255 (59h21m17.705423295s) [noise=true , address=79.127.220.97:4361 , piece_id=ZEPO7TFYGBAYVJ4HOAVUQYT5FXIHXC67K7WVM7I46U5FPWHQ35TQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:09.417611896 (59h21m19.140174936s) [noise=true , address=79.127.220.101:4048 , piece_id=XLICLHKFI5LQZZUGMARSQH7D63ZD6UVFPZQNT6UPI2CGWQ6HKWKA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:18.284819173 (59h21m28.007382213s) [noise=true , address=77.90.31.15:4722 , piece_id=N3AHIRSEAGBNUQHURAJRH5HYHRC4H4QAGFPLDPKUM2WU65XUVTHA, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:25.111931043 (59h21m34.834494083s) [noise=true , address=45.91.116.8:4065 , piece_id=ZFC5NG6WZ5XWVGJ4IAYMX4P3NO3XW722YCQJBPNTRAMKLPKSK3VQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:57:58.409766678 (59h22m8.132329708s) [noise=true , address=77.90.31.10:4523 , piece_id=4MHJ7S7PVPSLORYVSIQDBMYHDCHOFQHDBDAUEH3B3VL5ZIZNEE2Q, seg_pos={0 0}, upload=56951]: Done uploading piece. err: <nil> Jul 26 22:58:02.254205730 (59h22m11.97676879s) [noise=true , address=77.90.31.10:4548 , piece_id=UZVX3Z7UPKDUCAAGFHHVGXLA4FPLEKYISWE7PIC5PJDEA5MY5EEQ, seg_pos={0 0}, upload=56951]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:58:02.254261236 (59h22m11.976824317s) [seg_pos={0 0}, upload=56951]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:58:02.487970969 (59h22m12.210534029s) [upload=56951]: Done uploading segments... Jul 26 22:58:02.487995947 (59h22m12.210558977s) [upload=56951]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.235974946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X..........X.X........X.X.................] duration: 5.68456786s FAIL (2025-07-26 22:57:59 UTC) orbiter warning: uplink: too many requests Duration: 5.68459374s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.........X........] duration: 5.798489062s FAIL (2025-07-26 22:57:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.798518779s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [................X...X...............X..XX...X.X.X.X.......XX] duration: 1m0.000873327s FAIL (2025-07-26 22:57:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000883966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X..........X.X........X.X................] duration: 5.714469096s FAIL (2025-07-26 22:57:12 UTC) orbiter warning: uplink: too many requests Duration: 5.714520714s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXX] duration: 1m0.230904702s FAIL (2025-07-26 22:57:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9h20m12.60075917s) [seg_pos={0 0}, upload=56936]: Begin upload segment... Jul 26 22:56:02.878256135 (59h20m12.600819185s) [seg_pos={0 0}, upload=56936]: Done begin upload segment. Jul 26 22:56:02.878270753 (59h20m12.600833793s) [noise=true , address=79.127.219.48:4090 , piece_id=ULU4QDOMHLZQ7I4Y4F2ASQNGNGTR22KP3H6PIMWOVCCBSPJVFR7Q, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.878276914 (59h20m12.600839954s) [noise=true , address=79.127.219.38:4005 , piece_id=4BO7TGTQ7RKGPOMZOKOHGIPBVYVHV6MHACHY3UCJERN3RDMQ7WOQ, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.878677060 (59h20m12.6012401s) [noise=true , address=45.91.116.9:4288 , piece_id=5YCNF3MMIZZKY2STFOBJLCTP7U7PMYZBPDSS36BK7EAGVY2UMT4Q, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879010578 (59h20m12.601573628s) [noise=true , address=77.90.31.13:4639 , piece_id=JNXXXCFH5SQ3E7JWHIHP7IM6T577S66IMBLNMAXNZCVP6PIP6JKQ, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879072497 (59h20m12.601635537s) [noise=true , address=79.127.205.235:4662 , piece_id=RT22YEE2MAEG7Z6ICN4VFRFM7CH5WSCH5JK6THTDSXSXI7KMYT4A, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879342964 (59h20m12.601906004s) [noise=true , address=79.127.219.38:4004 , piece_id=K3RPE72AQ5RW25RIM2FM6G22ZGNMXXCIX5OEL4W2HXWCBAX5C2QA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879366860 (59h20m12.6019299s) [noise=true , address=156.146.43.225:4448 , piece_id=TF7PLIYXVNFD4CSHV5LHZHYM33SA5OSAVSLSIAAFNEJZM7MNMRVQ, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879617369 (59h20m12.602180409s) [noise=true , address=79.127.205.233:4630 , piece_id=WB3V6JOK67MAT7CWLPXVIHWR3HJYKTRF5PK6QONZP3VODI3N3KCA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.879640133 (59h20m12.602203173s) [noise=true , address=77.90.31.18:4832 , piece_id=OXJGV3PIW4DLF6DHEWJU2RFDQOF6PNUVKVB6K7H57A7BXCY4AKEA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.880118017 (59h20m12.602681067s) [noise=true , address=45.91.116.14:4194 , piece_id=M6H6JVCZG2XKXGC3IQONAKTUA26643646COE2Z23T5OT636UJ47A, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.880458458 (59h20m12.603021498s) [noise=true , address=79.127.205.245:4221 , piece_id=X6CB6KAN7FR6CGXVLPZTJP4CY5KZRND4XBQLF6J5PQT3LJWVDSJQ, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.880658331 (59h20m12.603221371s) [noise=true , address=77.90.31.17:4796 , piece_id=RUUNIE7WFKYOCUNSQJICCQTOMLBTMCCSSRGA6MUGY6WVYHV3THMA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.880979475 (59h20m12.603542515s) [noise=true , address=79.127.201.215:4350 , piece_id=3SWUWK4UQTZ66QCH6X5JRQO2P3E3J7LPMFYN2NHVKPSS2FH6CHGQ, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.881159409 (59h20m12.60372245s) [noise=true , address=79.127.220.101:4048 , piece_id=BVUW47H4G3GVQEWCRKCYLFOAF37L76U2PJ4JJ7CYYB6QIJOQKTTA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.881419857 (59h20m12.603982897s) [noise=true , address=77.90.31.19:4871 , piece_id=SHF4DTHG6YVKNLF24ZM5E2W523JIUD334WEJ75QDNYRKNREQ7OJA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.881604581 (59h20m12.604167621s) [noise=true , address=77.90.31.10:4543 , piece_id=MYQN3ACXQJNNSYRNKRPYDGDLSVJ7LKXVBDUQ5SK7EI4OJG2SBXIA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.881949370 (59h20m12.60451241s) [noise=true , address=77.90.31.11:4564 , piece_id=KIOBRSGDIS4OR6XZ7EMG7MITD6RE6AXRIAWQKTV7T46P4WDHUO7A, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:02.882235699 (59h20m12.604798749s) [noise=true , address=77.90.31.21:4896 , piece_id=HTBYSXXIV2APEQP7Y422NJKMWIK4OTEL6GF76XTSMS6O7HSQRZAA, seg_pos={0 0}, upload=56936]: Uploading piece... Jul 26 22:56:03.991546813 (59h20m13.714109853s) [noise=true , address=77.90.31.19:4871 , piece_id=SHF4DTHG6YVKNLF24ZM5E2W523JIUD334WEJ75QDNYRKNREQ7OJA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:03.991905359 (59h20m13.714468399s) [upload=56936]: Next returned nil segment Jul 26 22:56:03.991911811 (59h20m13.714474851s) [upload=56936]: Waiting for error group managing segments... Jul 26 22:56:03.996642130 (59h20m13.71920517s) [noise=true , address=79.127.219.48:4090 , piece_id=ULU4QDOMHLZQ7I4Y4F2ASQNGNGTR22KP3H6PIMWOVCCBSPJVFR7Q, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:04.089917366 (59h20m13.812480406s) [noise=true , address=79.127.220.101:4048 , piece_id=BVUW47H4G3GVQEWCRKCYLFOAF37L76U2PJ4JJ7CYYB6QIJOQKTTA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:04.314508212 (59h20m14.037071252s) [noise=true , address=79.127.205.235:4662 , piece_id=RT22YEE2MAEG7Z6ICN4VFRFM7CH5WSCH5JK6THTDSXSXI7KMYT4A, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:04.331887621 (59h20m14.054450661s) [noise=true , address=79.127.205.233:4630 , piece_id=WB3V6JOK67MAT7CWLPXVIHWR3HJYKTRF5PK6QONZP3VODI3N3KCA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:04.475923227 (59h20m14.198486267s) [noise=true , address=79.127.205.245:4221 , piece_id=X6CB6KAN7FR6CGXVLPZTJP4CY5KZRND4XBQLF6J5PQT3LJWVDSJQ, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:04.506961176 (59h20m14.229524206s) [noise=true , address=156.146.43.225:4448 , piece_id=TF7PLIYXVNFD4CSHV5LHZHYM33SA5OSAVSLSIAAFNEJZM7MNMRVQ, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:04.856582939 (59h20m14.57914598s) [noise=true , address=79.127.219.38:4004 , piece_id=K3RPE72AQ5RW25RIM2FM6G22ZGNMXXCIX5OEL4W2HXWCBAX5C2QA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:04.902422276 (59h20m14.624985316s) [noise=true , address=77.90.31.21:4896 , piece_id=HTBYSXXIV2APEQP7Y422NJKMWIK4OTEL6GF76XTSMS6O7HSQRZAA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:04.963285584 (59h20m14.685848624s) [noise=true , address=77.90.31.11:4564 , piece_id=KIOBRSGDIS4OR6XZ7EMG7MITD6RE6AXRIAWQKTV7T46P4WDHUO7A, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:05.175037909 (59h20m14.897600939s) [noise=true , address=79.127.219.38:4005 , piece_id=4BO7TGTQ7RKGPOMZOKOHGIPBVYVHV6MHACHY3UCJERN3RDMQ7WOQ, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:15.122758447 (59h20m24.845321478s) [noise=true , address=45.91.116.9:4288 , piece_id=5YCNF3MMIZZKY2STFOBJLCTP7U7PMYZBPDSS36BK7EAGVY2UMT4Q, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:56:18.276757085 (59h20m27.999320125s) [noise=true , address=45.91.116.14:4194 , piece_id=M6H6JVCZG2XKXGC3IQONAKTUA26643646COE2Z23T5OT636UJ47A, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:56.121297943 (59h21m5.843860983s) [noise=true , address=79.127.201.215:4350 , piece_id=3SWUWK4UQTZ66QCH6X5JRQO2P3E3J7LPMFYN2NHVKPSS2FH6CHGQ, seg_pos={0 0}, upload=56936]: Done uploading piece. err: <nil> Jul 26 22:57:01.792793489 (59h21m11.515356529s) [noise=true , address=77.90.31.10:4543 , piece_id=MYQN3ACXQJNNSYRNKRPYDGDLSVJ7LKXVBDUQ5SK7EI4OJG2SBXIA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:57:01.792821091 (59h21m11.515384141s) [noise=true , address=77.90.31.13:4639 , piece_id=JNXXXCFH5SQ3E7JWHIHP7IM6T577S66IMBLNMAXNZCVP6PIP6JKQ, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:57:01.792892999 (59h21m11.515456049s) [noise=true , address=77.90.31.17:4796 , piece_id=RUUNIE7WFKYOCUNSQJICCQTOMLBTMCCSSRGA6MUGY6WVYHV3THMA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 188.166.225.170:53802->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:01.793020523 (59h21m11.515583563s) [noise=true , address=77.90.31.18:4832 , piece_id=OXJGV3PIW4DLF6DHEWJU2RFDQOF6PNUVKVB6K7H57A7BXCY4AKEA, seg_pos={0 0}, upload=56936]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: read tcp 188.166.225.170:39596->77.90.31.18:4832: use of closed network connection; context deadline exceeded Jul 26 22:57:01.793049017 (59h21m11.515612057s) [seg_pos={0 0}, upload=56936]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:57:02.022378330 (59h21m11.74494137s) [upload=56936]: Done uploading segments... Jul 26 22:57:02.022399651 (59h21m11.744962691s) [upload=56936]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230929259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration: 5.732515393s FAIL (2025-07-26 22:56:53 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.732543948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X..........X.X........X.X...............] duration: 5.654914452s FAIL (2025-07-26 22:56:36 UTC) orbiter warning: uplink: too many requests Duration: 5.654939051s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................................X.........X......] duration: 5.792011992s FAIL (2025-07-26 22:56:24 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.792036578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.................X...X...............X..XX...X.X.X.X.......X] duration: 1m0.000620692s FAIL (2025-07-26 22:56:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000631462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XX] duration: 1m0.237756032s FAIL (2025-07-26 22:55:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 434 , piece_id=TA56TTQIC5MOFTPLMBYRHPR25NXDJSMIDSA5ZRCAA45YB3VKYRWQ, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.956497908 (59h19m9.679060938s) [noise=true , address=77.90.31.11:4585 , piece_id=U3SSVJAQXSR5SF5WKSXKQA6VVIAWRO6ZGURSF457266ZDPGEJCSA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.956607488 (59h19m9.679170528s) [noise=true , address=79.127.205.234:4614 , piece_id=4KABCUYJ2GJMJ5HHKO4HPKNRKFRHDNCEDN22QB3TLSGHNSPMXYSA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.956767454 (59h19m9.679330494s) [noise=true , address=77.90.31.16:4746 , piece_id=IOCGEUP5I7U2G32LJXWZSTOBBNYZEXYHUTELJ4IWFJSH6TTRN5KQ, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.956908394 (59h19m9.679471434s) [noise=true , address=64.49.13.140:4508 , piece_id=QHZB7UXIYRA2DNQTJL2LO7S3YP52Y2C7PLXPLI6QE7OOP3RUCJQA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.956988788 (59h19m9.679551828s) [noise=true , address=79.127.205.246:4273 , piece_id=3QF74YCAJ4MZBJYTJAV5B27X3OQOZSW5RTFSZHQBD3MGELIYFOFA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.957159484 (59h19m9.679722524s) [noise=true , address=45.91.116.14:4206 , piece_id=XMZEPCHCMMMEQ4EUWBVLC3DH4AUDHKMTN5UL4CV42X2BHNDVSGMQ, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.957318518 (59h19m9.679881548s) [noise=true , address=64.49.13.138:4483 , piece_id=EGN2PDU4WRHQFJCLDMJBO4RK5CWQPYBEM3ALNPTXLWLC2TE2Y24A, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:54:59.957553007 (59h19m9.680116047s) [noise=true , address=79.127.201.215:4356 , piece_id=DGIX3S7GIPWVCPWXKY7IIIO2JMYNHS2Y7M6NRI74IR6UWQBSGLIA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:00.939508627 (59h19m10.662071667s) [noise=true , address=64.49.13.140:4508 , piece_id=QHZB7UXIYRA2DNQTJL2LO7S3YP52Y2C7PLXPLI6QE7OOP3RUCJQA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:00.992617740 (59h19m10.71518077s) [noise=true , address=156.146.43.225:4434 , piece_id=TA56TTQIC5MOFTPLMBYRHPR25NXDJSMIDSA5ZRCAA45YB3VKYRWQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:01.016921627 (59h19m10.739484667s) [noise=true , address=77.90.31.16:4746 , piece_id=IOCGEUP5I7U2G32LJXWZSTOBBNYZEXYHUTELJ4IWFJSH6TTRN5KQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:01.030230829 (59h19m10.752793859s) [noise=true , address=79.127.219.47:4061 , piece_id=6WAWPO4F4Q6QPWTEZSHOKVAFTJW37M4SA5QRBLRQ3GWVQICP53SQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:01.058870159 (59h19m10.781433199s) [upload=56919]: Next returned nil segment Jul 26 22:55:01.058884757 (59h19m10.781447797s) [upload=56919]: Waiting for error group managing segments... Jul 26 22:55:01.062682923 (59h19m10.785245963s) [noise=true , address=64.49.13.138:4483 , piece_id=EGN2PDU4WRHQFJCLDMJBO4RK5CWQPYBEM3ALNPTXLWLC2TE2Y24A, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:01.346884301 (59h19m11.069447352s) [noise=true , address=79.127.205.233:4650 , piece_id=ENTKRKF7TSGV6VSYNFKTWO7MV75DGSDZOY6CQJHXFO42IBB4EFLQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.354262477 (59h19m11.076825527s) [noise=true , address=79.127.205.245:4251 , piece_id=M37PEDIZQPV2RIVIUAD5UZXS5TQGAZUIGBZ6VOS2PH22E3YKFDSA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.470049471 (59h19m11.192612511s) [noise=true , address=79.127.205.246:4273 , piece_id=3QF74YCAJ4MZBJYTJAV5B27X3OQOZSW5RTFSZHQBD3MGELIYFOFA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.479241636 (59h19m11.201804686s) [noise=true , address=79.127.205.233:4649 , piece_id=XUS7EXEITEI46ZFKPYOT5HHOP5H2RY6ANE5IGXGWMXJYJG6I7SPA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.525085591 (59h19m11.247648641s) [noise=true , address=79.127.205.234:4614 , piece_id=4KABCUYJ2GJMJ5HHKO4HPKNRKFRHDNCEDN22QB3TLSGHNSPMXYSA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.815082107 (59h19m11.537645157s) [noise=true , address=79.127.219.38:4004 , piece_id=7XNV7BRG2ZU445JK5LP5WSANSD7S627KWOTOYRHOFTOL24WDHRLQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:01.903627934 (59h19m11.626190984s) [noise=true , address=77.90.31.11:4585 , piece_id=U3SSVJAQXSR5SF5WKSXKQA6VVIAWRO6ZGURSF457266ZDPGEJCSA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:03.229932343 (59h19m12.952495383s) [noise=true , address=64.49.13.170:4719 , piece_id=RC3TZFC7RSLA7GTXOBMNM745V4HVVV2P4KTWZNYWRNTQVQOFW5EA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:11.416599602 (59h19m21.139162642s) [noise=true , address=45.91.116.11:4124 , piece_id=ZN6CAVOA4J3WRO6E77EMP6GAK7XWWQH2KCKCNUXP4OI3QLKH6UCQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:15.283308607 (59h19m25.005871647s) [noise=true , address=77.90.31.15:4712 , piece_id=UGHP2JQB7YUNPRHUSCR6XYN6IKXYQV2F4KM64HRXYH6BR3UYCIOQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:16.653175533 (59h19m26.375738573s) [noise=true , address=45.91.116.14:4206 , piece_id=XMZEPCHCMMMEQ4EUWBVLC3DH4AUDHKMTN5UL4CV42X2BHNDVSGMQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:39.291106803 (59h19m49.013669853s) [noise=true , address=45.91.116.14:4194 , piece_id=N3N6R7WEOSD2RMTEXJ4XU5UIACHPE5HF4FUT7NUN6I3HQJWYXTJQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:55.044013286 (59h20m4.766576326s) [noise=true , address=79.127.201.215:4356 , piece_id=DGIX3S7GIPWVCPWXKY7IIIO2JMYNHS2Y7M6NRI74IR6UWQBSGLIA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: <nil> Jul 26 22:55:55.500749824 (59h20m5.223312864s) [noise=true , address=79.127.219.51:4212 , piece_id=L4HHJWO7V6GBWGT5PJH4JHMIHLLJPEQOWCPW2X4QN5HFXD7M6P4A, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:55.500775293 (59h20m5.223338333s) [noise=true , address=156.146.43.225:4448 , piece_id=N4UUG5KPHNTXLG7QIU7J23RHAXF77N5QXOJAWLIJ4GXQIM7N2UUA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:55.500832923 (59h20m5.223395963s) [noise=true , address=45.91.116.13:4175 , piece_id=VL4WRFE6BW2YHJUE2445H3PA2QWCLHSRUSFKKEAKEN6GGHKMRNBA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:55.501235963 (59h20m5.223799003s) [noise=true , address=45.91.116.5:4310 , piece_id=4PKCGMWDBDJFN4HLX3CRA7XR55HO4H4467HGNZTSQJLULMJ4CTYQ, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:55.501614207 (59h20m5.224177247s) [noise=true , address=79.127.201.215:4350 , piece_id=AI4BQCFNCRKXR5FUFHZXF4B4XYNPAKGNMDV7FLXKQSVDB3LANRCA, seg_pos={0 0}, upload=56919]: Uploading piece... Jul 26 22:55:56.516216953 (59h20m6.238779993s) [noise=true , address=156.146.43.225:4448 , piece_id=N4UUG5KPHNTXLG7QIU7J23RHAXF77N5QXOJAWLIJ4GXQIM7N2UUA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:56.631612779 (59h20m6.35417582s) [noise=true , address=79.127.219.51:4212 , piece_id=L4HHJWO7V6GBWGT5PJH4JHMIHLLJPEQOWCPW2X4QN5HFXD7M6P4A, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:58.843086056 (59h20m8.565649106s) [noise=true , address=45.91.116.13:4175 , piece_id=VL4WRFE6BW2YHJUE2445H3PA2QWCLHSRUSFKKEAKEN6GGHKMRNBA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:55:58.843114670 (59h20m8.56567771s) [noise=true , address=45.91.116.5:4310 , piece_id=4PKCGMWDBDJFN4HLX3CRA7XR55HO4H4467HGNZTSQJLULMJ4CTYQ, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:55:58.843188000 (59h20m8.56575104s) [noise=true , address=79.127.201.215:4350 , piece_id=AI4BQCFNCRKXR5FUFHZXF4B4XYNPAKGNMDV7FLXKQSVDB3LANRCA, seg_pos={0 0}, upload=56919]: Done uploading piece. err: ecclient: upload failed (node:1WhZAWvHCTj5xVkUuVZtgkFB1dhFKpBPq18YMKkepFx5qxkUAr, address:79.127.201.215:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:55:58.843217797 (59h20m8.565780837s) [seg_pos={0 0}, upload=56919]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:55:59.079471748 (59h20m8.802034788s) [upload=56919]: Done uploading segments... Jul 26 22:55:59.079496275 (59h20m8.802059315s) [upload=56919]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.237769508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..........X.X........X.X..............] duration: 6.326053786s FAIL (2025-07-26 22:55:25 UTC) orbiter warning: uplink: too many requests Duration: 6.326071832s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................................X.........X.....] duration: 5.741440315s FAIL (2025-07-26 22:55:15 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.7414631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...X] duration: 1m0.23077703s FAIL (2025-07-26 22:54:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 8 (59h18m8.760718318s) [upload=56904]: Uploading segments... Jul 26 22:53:59.038462857 (59h18m8.761025937s) [upload=56904]: Got next segment. Inline: false Jul 26 22:53:59.038512903 (59h18m8.761076013s) [upload=56904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:53:59.038522842 (59h18m8.761085912s) [upload=56904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:53:59.496795228 (59h18m9.219358268s) [seg_pos={0 0}, upload=56904]: Begin upload segment... Jul 26 22:53:59.496843381 (59h18m9.219406421s) [seg_pos={0 0}, upload=56904]: Done begin upload segment. Jul 26 22:53:59.496862667 (59h18m9.219425707s) [noise=true , address=45.91.116.12:4229 , piece_id=6EZNDTS4YVBJZLUWIKUBWMUQV3CFBSQQOVXLQDXOKSGDJQW3ELMQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.496896973 (59h18m9.219460013s) [noise=true , address=45.91.116.11:4151 , piece_id=OH2F5WFB2EN4TMU6ANO5ABSAO5R6PPYQ7CVMIKUY2CTNU5ZT5EHQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497209320 (59h18m9.21977236s) [noise=true , address=77.90.31.12:4624 , piece_id=ZHICVBXFWWV3HPJQPRZYRTIH3YJRN3MW3ROC4QBGMNEPNXBZPOPA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497290065 (59h18m9.219853105s) [noise=true , address=64.49.13.26:4951 , piece_id=U47GCDUTDNP4OTO52JNXQ5RMBUPKL6RJX3GENJFXR7ZCZWCWMZ5Q, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497466763 (59h18m9.220029803s) [noise=true , address=45.91.116.13:4175 , piece_id=FKQEV5HCQ5HU3ZV24VFUXY3324MGXLMZIY7PSFTEQSVFXYJHXCSA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497560553 (59h18m9.220123583s) [noise=true , address=64.49.13.140:4515 , piece_id=PCDYM6M4VWROIX42ELKD22PKG7YV3H7QTDHJ6VYGSXGIXVDMN4TA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497717432 (59h18m9.220280462s) [noise=true , address=79.127.219.51:4212 , piece_id=AX4RYOK4GC2S3IL2RMG2JNT4TORDNSZTIZQ4FPYQO7AAOHB6IVHA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497834787 (59h18m9.220397817s) [noise=true , address=79.127.205.233:4650 , piece_id=ZQ2QWTU7XG2CFJ7D4XJ2BPCIOVY7LR25VIKHMOCUS6G343ATIVPQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.497995474 (59h18m9.220558514s) [noise=true , address=77.90.31.11:4579 , piece_id=ZY2F2YRYXTMAC4ZRCWPQUGQ4MWFEJAY4BW4YLEEYZKATKNEFDHWQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.498216617 (59h18m9.220779657s) [noise=true , address=79.127.201.215:4343 , piece_id=5QYVI2EP2F2WK6BWTTI6JVRUP5NV5CD6H3V73ANQOCKGALTMKFLA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.498424886 (59h18m9.220987926s) [noise=true , address=79.127.219.47:4048 , piece_id=CUYEU3Y7DZWSL36E3GO5M3TNA6DSEKBTONT6M4DIGXRVCTGLAMWQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.498636912 (59h18m9.221199952s) [noise=true , address=79.127.205.245:4251 , piece_id=7HOVV27TSED34JZLHLTXARL3ASR3TO2OPDM4QYZ66H43BDJ5HW3Q, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.498843156 (59h18m9.221406196s) [noise=true , address=79.127.205.246:4261 , piece_id=I2NQZIZI5SP24RI3WPNBWBYB4C2HMT5HUN6VTAFFGBLFSGR47GLQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.499049500 (59h18m9.22161254s) [noise=true , address=77.90.31.13:4634 , piece_id=YNOKYYRRJ5TSBQYHGUTVZIXZBVY6QJWBECE5IKQYMXJF62JNIDCQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.499227091 (59h18m9.221790121s) [noise=true , address=45.91.116.14:4206 , piece_id=AQYBUZE3DRVJVQPNTTFPX6VDECBQ2SB5BVIILBI6RXXTMJRGPZRQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.499418888 (59h18m9.221981928s) [noise=true , address=45.91.116.5:4310 , piece_id=V56XCHOUG2RAIMXFFBAQQMYWHE5IHDI2PP67Y3GLVBN3XOEYJJDA, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.499598962 (59h18m9.222162002s) [noise=true , address=156.146.43.225:4434 , piece_id=L4LXSJR7NCVTGCWKV3KEAZNKR2T2X36YFXZCFETNYUJKUZA373PQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:53:59.499783224 (59h18m9.222346265s) [noise=true , address=64.49.13.138:4482 , piece_id=TZPBXHDDKPE3KF3LNQX237PVAJDNX267LVPKV7LSUQBQ3LYTFPUQ, seg_pos={0 0}, upload=56904]: Uploading piece... Jul 26 22:54:00.364607027 (59h18m10.087170067s) [noise=true , address=64.49.13.140:4515 , piece_id=PCDYM6M4VWROIX42ELKD22PKG7YV3H7QTDHJ6VYGSXGIXVDMN4TA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:00.518031044 (59h18m10.240594084s) [noise=true , address=45.91.116.11:4151 , piece_id=OH2F5WFB2EN4TMU6ANO5ABSAO5R6PPYQ7CVMIKUY2CTNU5ZT5EHQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:00.579204396 (59h18m10.301767436s) [noise=true , address=79.127.219.47:4048 , piece_id=CUYEU3Y7DZWSL36E3GO5M3TNA6DSEKBTONT6M4DIGXRVCTGLAMWQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:00.607971711 (59h18m10.330534741s) [upload=56904]: Next returned nil segment Jul 26 22:54:00.607989816 (59h18m10.330552856s) [upload=56904]: Waiting for error group managing segments... Jul 26 22:54:00.887260048 (59h18m10.609823088s) [noise=true , address=79.127.205.233:4650 , piece_id=ZQ2QWTU7XG2CFJ7D4XJ2BPCIOVY7LR25VIKHMOCUS6G343ATIVPQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:00.914318794 (59h18m10.636881864s) [noise=true , address=79.127.205.245:4251 , piece_id=7HOVV27TSED34JZLHLTXARL3ASR3TO2OPDM4QYZ66H43BDJ5HW3Q, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:00.919258454 (59h18m10.641821495s) [noise=true , address=79.127.205.246:4261 , piece_id=I2NQZIZI5SP24RI3WPNBWBYB4C2HMT5HUN6VTAFFGBLFSGR47GLQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:00.958292501 (59h18m10.680855541s) [noise=true , address=64.49.13.26:4951 , piece_id=U47GCDUTDNP4OTO52JNXQ5RMBUPKL6RJX3GENJFXR7ZCZWCWMZ5Q, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:01.518444705 (59h18m11.241007745s) [noise=true , address=77.90.31.11:4579 , piece_id=ZY2F2YRYXTMAC4ZRCWPQUGQ4MWFEJAY4BW4YLEEYZKATKNEFDHWQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:04.260750811 (59h18m13.983313841s) [noise=true , address=156.146.43.225:4434 , piece_id=L4LXSJR7NCVTGCWKV3KEAZNKR2T2X36YFXZCFETNYUJKUZA373PQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:04.843266669 (59h18m14.565829709s) [noise=true , address=45.91.116.14:4206 , piece_id=AQYBUZE3DRVJVQPNTTFPX6VDECBQ2SB5BVIILBI6RXXTMJRGPZRQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:05.268877951 (59h18m14.991440991s) [noise=true , address=79.127.219.51:4212 , piece_id=AX4RYOK4GC2S3IL2RMG2JNT4TORDNSZTIZQ4FPYQO7AAOHB6IVHA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:05.786008798 (59h18m15.508571828s) [noise=true , address=45.91.116.12:4229 , piece_id=6EZNDTS4YVBJZLUWIKUBWMUQV3CFBSQQOVXLQDXOKSGDJQW3ELMQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:07.167020245 (59h18m16.889583286s) [noise=true , address=64.49.13.138:4482 , piece_id=TZPBXHDDKPE3KF3LNQX237PVAJDNX267LVPKV7LSUQBQ3LYTFPUQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:11.025810744 (59h18m20.748373774s) [noise=true , address=45.91.116.13:4175 , piece_id=FKQEV5HCQ5HU3ZV24VFUXY3324MGXLMZIY7PSFTEQSVFXYJHXCSA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:12.614170832 (59h18m22.336733922s) [noise=true , address=77.90.31.13:4634 , piece_id=YNOKYYRRJ5TSBQYHGUTVZIXZBVY6QJWBECE5IKQYMXJF62JNIDCQ, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:14.500413676 (59h18m24.222976706s) [noise=true , address=45.91.116.5:4310 , piece_id=V56XCHOUG2RAIMXFFBAQQMYWHE5IHDI2PP67Y3GLVBN3XOEYJJDA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:46.620853265 (59h18m56.343416305s) [noise=true , address=79.127.201.215:4343 , piece_id=5QYVI2EP2F2WK6BWTTI6JVRUP5NV5CD6H3V73ANQOCKGALTMKFLA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: <nil> Jul 26 22:54:58.394445207 (59h19m8.117008247s) [noise=true , address=77.90.31.12:4624 , piece_id=ZHICVBXFWWV3HPJQPRZYRTIH3YJRN3MW3ROC4QBGMNEPNXBZPOPA, seg_pos={0 0}, upload=56904]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: read tcp 188.166.225.170:39520->77.90.31.12:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:54:58.394467309 (59h19m8.11703034s) [seg_pos={0 0}, upload=56904]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:54:58.623589616 (59h19m8.346152656s) [upload=56904]: Done uploading segments... Jul 26 22:54:58.623615787 (59h19m8.346178817s) [upload=56904]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.230799343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................X...X...............X..XX...X.X.X.X.......] duration: 23.989439942s FAIL (2025-07-26 22:54:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.989459199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................................X.........X....] duration: 7.002346513s FAIL (2025-07-26 22:54:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 7.002371501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X..........X.X........X.X.............] duration: 5.842647972s FAIL (2025-07-26 22:54:21 UTC) orbiter warning: uplink: too many requests Duration: 5.842676267s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................X...X...............X..XX...X.X.X.X......] duration: 27.047540975s FAIL (2025-07-26 22:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.047551726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................................X.........X...] duration: 5.741973988s FAIL (2025-07-26 22:54:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.741999557s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....................X...X...............X..XX...X.X.X.X.....] duration: 23.957878882s FAIL (2025-07-26 22:53:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.957897838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X..........................................X.........X..] duration: 5.788363984s FAIL (2025-07-26 22:53:50 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 5.788390075s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....................X...X...............X..XX...X.X.X.X....] duration: 9.512830044s FAIL (2025-07-26 22:53:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.512850634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...X..........................................X.........X.] duration: 6.965305355s FAIL (2025-07-26 22:53:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 6.965325202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X...X..........................................X.........X] duration: 30.001484728s FAIL (2025-07-26 22:53:09 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== GKIELSB3YMQLTEWCKFWAZ23DIUY4DGZU6OHMZSMNSJYDY7KA, seg_pos={0 0}, upload=56859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Jul 26 22:51:32.574419840 (59h15m42.29698288s) [seg_pos={0 0}, upload=56859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:51:32.574496717 (59h15m42.297059757s) [upload=56859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:51:32.574501216 (59h15m42.297064256s) [upload=56859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:51:32.867907234 (59h15m42.590470294s) [upload=56859]: Done uploading segments... Jul 26 22:51:32.867931440 (59h15m42.59049449s) [upload=56859]: Upload finished. err: <nil> Jul 26 22:52:39.924544445 (59h16m49.647107485s) [upload=56882]: Starting upload Jul 26 22:52:39.924622936 (59h16m49.647185976s) [upload=56882]: Done starting upload Jul 26 22:52:39.924691927 (59h16m49.647254967s) [upload=56882]: Uploading segments... Jul 26 22:52:39.924883864 (59h16m49.647446904s) [upload=56882]: Got next segment. Inline: false Jul 26 22:52:39.924924953 (59h16m49.647487984s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:39.924930033 (59h16m49.647493073s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:40.370914310 (59h16m50.09347735s) [seg_pos={0 0}, upload=56882]: Begin upload segment... Jul 26 22:52:40.370995275 (59h16m50.093558315s) [seg_pos={0 0}, upload=56882]: Done begin upload segment. Jul 26 22:52:40.371031435 (59h16m50.093594475s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371591616 (59h16m50.094154686s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.371732245 (59h16m50.094295285s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372055123 (59h16m50.094618163s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372144845 (59h16m50.094707885s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372383691 (59h16m50.094946731s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372472702 (59h16m50.095035742s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372691199 (59h16m50.095254239s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372758288 (59h16m50.095321328s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.372948903 (59h16m50.095511943s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373168814 (59h16m50.095731854s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373233317 (59h16m50.095796357s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373473036 (59h16m50.096036076s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373519143 (59h16m50.096082193s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.373886306 (59h16m50.096449346s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:40.374185187 (59h16m50.096748227s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Uploading piece... Jul 26 22:52:42.108827923 (59h16m51.831390953s) [upload=56882]: Next returned nil segment Jul 26 22:52:42.108846850 (59h16m51.83140989s) [upload=56882]: Waiting for error group managing segments... Jul 26 22:52:42.394034815 (59h16m52.116597855s) [noise=true , address=45.91.116.5:4297 , piece_id=3BEPWZCKXFN7E2NTJUBZ7EBWZD3FLZ5NNUEYOFJG7J62VYDR75XQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.407267220 (59h16m52.12983026s) [noise=true , address=45.91.116.7:4009 , piece_id=VJEV6OVJ36OP7H7P27PI2CVGOLCW5WQWE7PKILXGXAGA3JZENY4A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.413625414 (59h16m52.136188454s) [noise=true , address=45.91.116.11:4117 , piece_id=OD2JCEAFGDYD4EVDAX766PYG5FDOSOSXJN5KXXCOZBDB4MFTU5VA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.423243875 (59h16m52.145806915s) [noise=true , address=45.91.116.9:4263 , piece_id=CYF6CGGTFZRRBPPDXKAAJFOGEIUQF6YNQNJVWBFNDQ32NAUWQ56A, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.468564970 (59h16m52.19112801s) [noise=true , address=45.91.116.12:4226 , piece_id=LYWVUEH3NXBE3YHLAFDW2IPBTPU7BRTQI5BXOEW7ALWEZHKMCPIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.470789616 (59h16m52.193352657s) [noise=true , address=77.90.31.14:4662 , piece_id=TP7SUQETGVZ3CJZUIQOQKEBQLJ2D6LKOIVAX6JP7GITF227XVOEQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.473509940 (59h16m52.19607298s) [noise=true , address=77.90.31.14:4663 , piece_id=WLLRHPSVDAEFOY7UDCAGHOUU63RQPKG42COIWLYPOXK6LR7JUM4Q, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.474056747 (59h16m52.196619787s) [noise=true , address=45.91.116.14:4190 , piece_id=B7X5CNG3BSWR3URUSXF3LQZDT42Z34SFJCPQMZIKZWQ7XNXPXYYA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.478597503 (59h16m52.201160533s) [noise=true , address=45.91.116.8:4048 , piece_id=X7YQDMS3VQN7RNCQIJ3X7DQ3HT5TQDLXXJVEOBM5AM5VGLHDDLRQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.485091357 (59h16m52.207654387s) [noise=true , address=45.91.116.4:4937 , piece_id=JU27G2W7OHW5ZORC37KAECGQSBGVQOSWENAOIB6DM7C57V2HBZSQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.486455276 (59h16m52.209018316s) [noise=true , address=45.91.116.6:4974 , piece_id=2SIZDPQ7KYMNY6LQESTYTTVJK65I6K4QZ27CEL2HMGMGVYZ2BNVA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:52:42.525569235 (59h16m52.248132265s) [noise=true , address=45.91.116.11:4119 , piece_id=RSG3YD67SAW334MJJRUQU2YBUEMZURXK5U4BQ2KRXER22WAV5HGA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093853545 (59h17m16.816416575s) [noise=true , address=77.90.31.19:4841 , piece_id=YZROEI2ISOMCIUJW5BRHVCYRS6AFB2WN73GMO57SORQDYTQC2LWA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: <nil> Jul 26 22:53:07.093878132 (59h17m16.816441172s) [seg_pos={0 0}, upload=56882]: Segment reached optimal threshold of 13 pieces. Jul 26 22:53:07.094063526 (59h17m16.816626566s) [noise=true , address=77.90.31.17:4770 , piece_id=MA2N44KTNDNNH3U7BJAPUTCAROHDA2LQOYCTWLRC36L6QJSLZQFA, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Jul 26 22:53:07.094231267 (59h17m16.816794297s) [noise=true , address=77.90.31.10:4517 , piece_id=UZAAUXRT4OTZC7INVY42KYZNQDIRUGTYP4AHDANNOQWYPYNBYBIQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54412->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094273088 (59h17m16.816836128s) [noise=true , address=77.90.31.18:4807 , piece_id=4DALGLRLNTRFX3TMM46GTMFYFOWV5ACBGK2OBUWP5V7NLJADV2IQ, seg_pos={0 0}, upload=56882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:44790->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:53:07.094308655 (59h17m16.816871695s) [seg_pos={0 0}, upload=56882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:53:07.094379541 (59h17m16.816942581s) [upload=56882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:53:07.094386534 (59h17m16.816949575s) [upload=56882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:53:07.381579838 (59h17m17.104142878s) [upload=56882]: Done uploading segments... Jul 26 22:53:07.381597993 (59h17m17.104161033s) [upload=56882]: Upload finished. err: <nil> Duration: 30.001507903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...] duration: 21.093502605s FAIL (2025-07-26 22:53:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== xt segment. Inline: false Jul 26 22:50:36.778825051 (59h14m46.501388091s) [upload=56847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:50:36.778830030 (59h14m46.50139307s) [upload=56847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:50:37.225887812 (59h14m46.948450852s) [seg_pos={0 0}, upload=56847]: Begin upload segment... Jul 26 22:50:37.225941515 (59h14m46.948504555s) [seg_pos={0 0}, upload=56847]: Done begin upload segment. Jul 26 22:50:37.225966172 (59h14m46.948529212s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226013422 (59h14m46.948576462s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226346440 (59h14m46.94890947s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226748188 (59h14m46.949311228s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227091294 (59h14m46.949654334s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227378654 (59h14m46.949941694s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227605669 (59h14m46.950168709s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227745877 (59h14m46.950308907s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228006906 (59h14m46.950569946s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228247677 (59h14m46.950810717s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229191712 (59h14m46.951754742s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229502658 (59h14m46.952065698s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229773405 (59h14m46.952336435s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230041388 (59h14m46.952604428s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230314962 (59h14m46.952878002s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230587393 (59h14m46.953150423s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230878059 (59h14m46.953441099s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.231150390 (59h14m46.95371342s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:38.316926460 (59h14m48.03948949s) [upload=56847]: Next returned nil segment Jul 26 22:50:38.316940838 (59h14m48.039503868s) [upload=56847]: Waiting for error group managing segments... Jul 26 22:50:38.401510759 (59h14m48.124073799s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.425985933 (59h14m48.148548963s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12RXve6Cd4pYRqUySbbaUKk9Xz3qjMPDAit9XRveV5DL5p3iVwU, address:79.127.201.215:4344): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.643607347 (59h14m48.366170387s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.648930621 (59h14m48.371493661s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.654551745 (59h14m48.377114795s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.130877690 (59h14m48.85344072s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.470512379 (59h14m49.193075459s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.750307248 (59h14m49.472870278s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:42.284433345 (59h14m52.006996395s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:43.411265568 (59h14m53.133828608s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:44.756669476 (59h14m54.479232516s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:50.217795148 (59h14m59.940358188s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:52.032395880 (59h15m1.75495892s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:22.835406175 (59h15m32.557969215s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:36.131904155 (59h15m45.854467185s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 188.166.225.170:42358->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132233846 (59h15m45.854796886s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 188.166.225.170:52946->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132488313 (59h15m45.855051363s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132865304 (59h15m45.855428344s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:56790->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132917745 (59h15m45.855480785s) [seg_pos={0 0}, upload=56847]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:51:36.368602746 (59h15m46.091165786s) [upload=56847]: Done uploading segments... Jul 26 22:51:36.368631772 (59h15m46.091194812s) [upload=56847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.093517745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..........X.X........X.X............] duration: 5.696906472s FAIL (2025-07-26 22:53:08 UTC) orbiter warning: uplink: too many requests Duration: 5.696935647s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......................X...X...............X..XX...X.X.X.X...] duration: 37.354041745s FAIL (2025-07-26 22:52:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.354065489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX..] duration: 39.531640373s FAIL (2025-07-26 22:52:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== xt segment. Inline: false Jul 26 22:50:36.778825051 (59h14m46.501388091s) [upload=56847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:50:36.778830030 (59h14m46.50139307s) [upload=56847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:50:37.225887812 (59h14m46.948450852s) [seg_pos={0 0}, upload=56847]: Begin upload segment... Jul 26 22:50:37.225941515 (59h14m46.948504555s) [seg_pos={0 0}, upload=56847]: Done begin upload segment. Jul 26 22:50:37.225966172 (59h14m46.948529212s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226013422 (59h14m46.948576462s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226346440 (59h14m46.94890947s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226748188 (59h14m46.949311228s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227091294 (59h14m46.949654334s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227378654 (59h14m46.949941694s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227605669 (59h14m46.950168709s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227745877 (59h14m46.950308907s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228006906 (59h14m46.950569946s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228247677 (59h14m46.950810717s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229191712 (59h14m46.951754742s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229502658 (59h14m46.952065698s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229773405 (59h14m46.952336435s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230041388 (59h14m46.952604428s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230314962 (59h14m46.952878002s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230587393 (59h14m46.953150423s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230878059 (59h14m46.953441099s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.231150390 (59h14m46.95371342s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:38.316926460 (59h14m48.03948949s) [upload=56847]: Next returned nil segment Jul 26 22:50:38.316940838 (59h14m48.039503868s) [upload=56847]: Waiting for error group managing segments... Jul 26 22:50:38.401510759 (59h14m48.124073799s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.425985933 (59h14m48.148548963s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12RXve6Cd4pYRqUySbbaUKk9Xz3qjMPDAit9XRveV5DL5p3iVwU, address:79.127.201.215:4344): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.643607347 (59h14m48.366170387s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.648930621 (59h14m48.371493661s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.654551745 (59h14m48.377114795s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.130877690 (59h14m48.85344072s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.470512379 (59h14m49.193075459s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.750307248 (59h14m49.472870278s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:42.284433345 (59h14m52.006996395s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:43.411265568 (59h14m53.133828608s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:44.756669476 (59h14m54.479232516s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:50.217795148 (59h14m59.940358188s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:52.032395880 (59h15m1.75495892s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:22.835406175 (59h15m32.557969215s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:36.131904155 (59h15m45.854467185s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 188.166.225.170:42358->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132233846 (59h15m45.854796886s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 188.166.225.170:52946->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132488313 (59h15m45.855051363s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132865304 (59h15m45.855428344s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:56790->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132917745 (59h15m45.855480785s) [seg_pos={0 0}, upload=56847]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:51:36.368602746 (59h15m46.091165786s) [upload=56847]: Done uploading segments... Jul 26 22:51:36.368631772 (59h15m46.091194812s) [upload=56847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 39.531668116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X..........X.X........X.X...........] duration: 5.769134918s FAIL (2025-07-26 22:52:04 UTC) orbiter warning: uplink: too many requests Duration: 5.769164423s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX.] duration: 23.823406513s FAIL (2025-07-26 22:52:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== xt segment. Inline: false Jul 26 22:50:36.778825051 (59h14m46.501388091s) [upload=56847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:50:36.778830030 (59h14m46.50139307s) [upload=56847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:50:37.225887812 (59h14m46.948450852s) [seg_pos={0 0}, upload=56847]: Begin upload segment... Jul 26 22:50:37.225941515 (59h14m46.948504555s) [seg_pos={0 0}, upload=56847]: Done begin upload segment. Jul 26 22:50:37.225966172 (59h14m46.948529212s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226013422 (59h14m46.948576462s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226346440 (59h14m46.94890947s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226748188 (59h14m46.949311228s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227091294 (59h14m46.949654334s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227378654 (59h14m46.949941694s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227605669 (59h14m46.950168709s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227745877 (59h14m46.950308907s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228006906 (59h14m46.950569946s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228247677 (59h14m46.950810717s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229191712 (59h14m46.951754742s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229502658 (59h14m46.952065698s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229773405 (59h14m46.952336435s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230041388 (59h14m46.952604428s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230314962 (59h14m46.952878002s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230587393 (59h14m46.953150423s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230878059 (59h14m46.953441099s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.231150390 (59h14m46.95371342s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:38.316926460 (59h14m48.03948949s) [upload=56847]: Next returned nil segment Jul 26 22:50:38.316940838 (59h14m48.039503868s) [upload=56847]: Waiting for error group managing segments... Jul 26 22:50:38.401510759 (59h14m48.124073799s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.425985933 (59h14m48.148548963s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12RXve6Cd4pYRqUySbbaUKk9Xz3qjMPDAit9XRveV5DL5p3iVwU, address:79.127.201.215:4344): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.643607347 (59h14m48.366170387s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.648930621 (59h14m48.371493661s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.654551745 (59h14m48.377114795s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.130877690 (59h14m48.85344072s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.470512379 (59h14m49.193075459s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.750307248 (59h14m49.472870278s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:42.284433345 (59h14m52.006996395s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:43.411265568 (59h14m53.133828608s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:44.756669476 (59h14m54.479232516s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:50.217795148 (59h14m59.940358188s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:52.032395880 (59h15m1.75495892s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:22.835406175 (59h15m32.557969215s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:36.131904155 (59h15m45.854467185s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 188.166.225.170:42358->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132233846 (59h15m45.854796886s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 188.166.225.170:52946->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132488313 (59h15m45.855051363s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132865304 (59h15m45.855428344s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:56790->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132917745 (59h15m45.855480785s) [seg_pos={0 0}, upload=56847]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:51:36.368602746 (59h15m46.091165786s) [upload=56847]: Done uploading segments... Jul 26 22:51:36.368631772 (59h15m46.091194812s) [upload=56847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.823423265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......................X...X...............X..XX...X.X.X.X..] duration: 16.046511754s FAIL (2025-07-26 22:51:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.046523427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..........X.X........X.X..........] duration: 5.810038451s FAIL (2025-07-26 22:51:45 UTC) orbiter warning: uplink: too many requests Duration: 5.810072828s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [........................X...X...............X..XX...X.X.X.X.] duration: 18.725125887s FAIL (2025-07-26 22:51:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.725137279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX] duration: 1m0.238538365s FAIL (2025-07-26 22:51:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== xt segment. Inline: false Jul 26 22:50:36.778825051 (59h14m46.501388091s) [upload=56847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:50:36.778830030 (59h14m46.50139307s) [upload=56847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:50:37.225887812 (59h14m46.948450852s) [seg_pos={0 0}, upload=56847]: Begin upload segment... Jul 26 22:50:37.225941515 (59h14m46.948504555s) [seg_pos={0 0}, upload=56847]: Done begin upload segment. Jul 26 22:50:37.225966172 (59h14m46.948529212s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226013422 (59h14m46.948576462s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226346440 (59h14m46.94890947s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.226748188 (59h14m46.949311228s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227091294 (59h14m46.949654334s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227378654 (59h14m46.949941694s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227605669 (59h14m46.950168709s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.227745877 (59h14m46.950308907s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228006906 (59h14m46.950569946s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.228247677 (59h14m46.950810717s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229191712 (59h14m46.951754742s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229502658 (59h14m46.952065698s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.229773405 (59h14m46.952336435s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230041388 (59h14m46.952604428s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230314962 (59h14m46.952878002s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230587393 (59h14m46.953150423s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.230878059 (59h14m46.953441099s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:37.231150390 (59h14m46.95371342s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Uploading piece... Jul 26 22:50:38.316926460 (59h14m48.03948949s) [upload=56847]: Next returned nil segment Jul 26 22:50:38.316940838 (59h14m48.039503868s) [upload=56847]: Waiting for error group managing segments... Jul 26 22:50:38.401510759 (59h14m48.124073799s) [noise=true , address=77.90.31.19:4875 , piece_id=UTNMZMO7G4AQWYGSLYFS5KHBAZ5YNEQ3W5P4X3HRERMCJCYLB6BQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.425985933 (59h14m48.148548963s) [noise=true , address=79.127.201.215:4344 , piece_id=QFE4Z2T44ZNMFOA4HTCYFX6UYESPGCVW2KTDNLKH4M2ASTTRSWYQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12RXve6Cd4pYRqUySbbaUKk9Xz3qjMPDAit9XRveV5DL5p3iVwU, address:79.127.201.215:4344): protocol: storage node overloaded, request limit: 2000 Jul 26 22:50:38.643607347 (59h14m48.366170387s) [noise=true , address=79.127.205.234:4603 , piece_id=KGGT6CXHJLCZUMTIMG7OJKGDX6BJYNEAYYZOEZKSRYCE6VOSUQTA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.648930621 (59h14m48.371493661s) [noise=true , address=79.127.205.246:4270 , piece_id=4YFYPEB3LFCAULYGB5KVZDXZKJ3ZXJCGAOAIYZFBLWKVO3J4XTUQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:38.654551745 (59h14m48.377114795s) [noise=true , address=79.127.205.233:4634 , piece_id=LLPWMBXPHNKENKBPH2FE6HI5IZWZO47A6NLDKID3BWHBVLL5Z3WQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.130877690 (59h14m48.85344072s) [noise=true , address=79.127.219.38:4003 , piece_id=X7RPWXH37FAM7IHQLNERS4WQ6H5XOVSBDCM4LDKMFIB66VFAA7HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.470512379 (59h14m49.193075459s) [noise=true , address=77.90.31.11:4573 , piece_id=NLJGH4LOIHFYMJKJNUJ3MHDGK4K7DZNORL7GQ7RCGCN6R3FXVZEA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:39.750307248 (59h14m49.472870278s) [noise=true , address=64.49.13.170:4719 , piece_id=5EOHYDG4Y6JOXQGLFD3K74NDW3Q5GYDSXTW3IVX7P5MCT6BKJUWA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:42.284433345 (59h14m52.006996395s) [noise=true , address=79.127.219.47:4061 , piece_id=CXKSQ6BSSVKT7BNY55CK5FRK6OH3AW7H3QP6IBDNF52HWCVLE5EA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:43.411265568 (59h14m53.133828608s) [noise=true , address=79.127.219.47:4048 , piece_id=P4J5NZWHVZIXOU7PX24LHLDFW6L7VKXCO27VBSVDAZWD5NFOB6UQ, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:44.756669476 (59h14m54.479232516s) [noise=true , address=64.49.13.138:4482 , piece_id=ABZ4QYBZCA4M5QQ74ZTCTCVLOR3NXYCP5ATIAU7SW7KA4MG3FMSA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:50.217795148 (59h14m59.940358188s) [noise=true , address=45.91.116.9:4288 , piece_id=FMGNMUJ6QRD7YPU4WY2WRPDNMZ4UTI6GWWOJHU2FVNMFIO4NAHCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:50:52.032395880 (59h15m1.75495892s) [noise=true , address=45.91.116.11:4151 , piece_id=FGQTJQ6ZATCESRUXTLSHFXZKHVALQEFVZCSY6KCM3M4C7XEX5IBA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:22.835406175 (59h15m32.557969215s) [noise=true , address=79.127.201.215:4351 , piece_id=LYTCFLIUL5WTIRPFPCQHYTBPQDVMZ3GJFIVKPBA5METXYOCTQKIA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: <nil> Jul 26 22:51:36.131904155 (59h15m45.854467185s) [noise=true , address=77.90.31.17:4790 , piece_id=US3RSR3RM6OLAH3ELG7RXRXOZCF5JSTNEON3ZDQOSE3SOTVARZ6Q, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 188.166.225.170:42358->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132233846 (59h15m45.854796886s) [noise=true , address=77.90.31.18:4832 , piece_id=Q5FSLSQ7G65A2IBFKBO4EKZJS3FXCJYYY4DM5LPSJTKVXYP4KCXA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 188.166.225.170:52946->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132488313 (59h15m45.855051363s) [noise=true , address=77.90.31.17:4800 , piece_id=NYSAUHTYBDFZBH5XWAMQPLDZULTQSNOHWPS3V2Q6SFLQ5OYF77HA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132865304 (59h15m45.855428344s) [noise=true , address=77.90.31.10:4535 , piece_id=73ILAQH3EWPYI6YCGWUXM67EASQBQNSNQ5YLZONRVVSL5BRSFYCA, seg_pos={0 0}, upload=56847]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:56790->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:51:36.132917745 (59h15m45.855480785s) [seg_pos={0 0}, upload=56847]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:51:36.368602746 (59h15m46.091165786s) [upload=56847]: Done uploading segments... Jul 26 22:51:36.368631772 (59h15m46.091194812s) [upload=56847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.238552874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X...X..........................................X.........] duration: 5.671948389s FAIL (2025-07-26 22:51:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.671984248s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)