Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.454093323s FAIL (2025-08-06 22:32:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 7.454115906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.898468924s FAIL (2025-08-06 22:32:31 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.898489232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............X.......................................] duration: 6.506623702s FAIL (2025-08-06 22:31:50 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.506655543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.246756053s FAIL (2025-08-06 22:31:50 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 2.246782935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.872543187s FAIL (2025-08-06 22:31:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.872567663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.583838448s FAIL (2025-08-06 22:31:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 8.583870919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.873580445s FAIL (2025-08-06 22:31:06 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.873612226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............X......................................] duration: 8.474783372s FAIL (2025-08-06 22:30:43 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 8.474802529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.429066433s FAIL (2025-08-06 22:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.429092995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.333403075s FAIL (2025-08-06 22:29:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.333429537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.050364695s FAIL (2025-08-06 22:29:53 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 2.050386146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.............X.....................................] duration: 6.297361637s FAIL (2025-08-06 22:29:38 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.297383088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.393560153s FAIL (2025-08-06 22:29:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.393584288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............X....................................] duration: 6.846236626s FAIL (2025-08-06 22:29:17 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.84626448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.868448807s FAIL (2025-08-06 22:29:04 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.868474456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.261093246s FAIL (2025-08-06 22:29:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.261118275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............X...................................] duration: 6.594324828s FAIL (2025-08-06 22:28:39 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.594348896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............X..................................] duration: 6.233330858s FAIL (2025-08-06 22:28:20 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.233355485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................................X...] duration: 1.925192874s FAIL (2025-08-06 22:27:58 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.925231678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.789060914s FAIL (2025-08-06 22:27:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.789087074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............X.................................] duration: 7.954220415s FAIL (2025-08-06 22:27:41 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 7.95424884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................................X..] duration: 1.889665371s FAIL (2025-08-06 22:27:39 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.889742059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 8.524698288s FAIL (2025-08-06 22:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 8.524723244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................................X.] duration: 1.973703289s FAIL (2025-08-06 22:27:17 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 1.973727836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................................X] duration: 858.374347ms FAIL (2025-08-06 22:26:55 UTC) orbiter warning: operation: Commit uplink: too many requests Duration: 858.393222ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.23481106s FAIL (2025-08-06 22:26:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.234834114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............X................................] duration: 6.645615452s FAIL (2025-08-06 22:26:41 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.645644728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.648463753s FAIL (2025-08-06 22:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.648489081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.846829593s FAIL (2025-08-06 22:25:50 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.846848599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.215591129s FAIL (2025-08-06 22:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.215619893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.............X...............................] duration: 10.574965663s FAIL (2025-08-06 22:25:36 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 10.574989066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.158626479s FAIL (2025-08-06 22:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.158644732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.030983337s FAIL (2025-08-06 22:25:19 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 3.031008584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.179080533s FAIL (2025-08-06 22:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.179109878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.............X..............................] duration: 6.722782184s FAIL (2025-08-06 22:24:34 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.72280565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.945063753s FAIL (2025-08-06 22:24:15 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.945087909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.............X.............................] duration: 8.162666279s FAIL (2025-08-06 22:24:14 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 8.162690324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.848037374s FAIL (2025-08-06 22:23:47 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.848060338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.138544437s FAIL (2025-08-06 22:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 6.138555958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001932829s FAIL (2025-08-06 22:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== VGIBWSTNBG3T5ZAPE5AQ, seg_pos={0 0}, upload=7839]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: context canceled; context canceled Aug 6 22:22:24.227847111 (8h13m32.16626296s) [seg_pos={0 0}, upload=7839]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:22:24.227953294 (8h13m32.166369183s) [upload=7839]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:22:24.227978793 (8h13m32.166394642s) [upload=7839]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:22:24.511796264 (8h13m32.450212103s) [upload=7839]: Done uploading segments... Aug 6 22:22:24.511838736 (8h13m32.450254585s) [upload=7839]: Upload finished. err: <nil> Aug 6 22:23:04.224450786 (8h14m12.162866635s) [upload=7849]: Starting upload Aug 6 22:23:04.224529026 (8h14m12.162944865s) [upload=7849]: Done starting upload Aug 6 22:23:04.224618757 (8h14m12.163034606s) [upload=7849]: Uploading segments... Aug 6 22:23:04.224730752 (8h14m12.163146591s) [upload=7849]: Got next segment. Inline: false Aug 6 22:23:04.224756230 (8h14m12.163172079s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 6 22:23:04.224760679 (8h14m12.163176528s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 6 22:23:04.456921276 (8h14m12.395337125s) [seg_pos={0 0}, upload=7849]: Begin upload segment... Aug 6 22:23:04.456981781 (8h14m12.395397631s) [seg_pos={0 0}, upload=7849]: Done begin upload segment. Aug 6 22:23:04.457004415 (8h14m12.395420264s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457046024 (8h14m12.395461873s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457558906 (8h14m12.395974755s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.457926018 (8h14m12.396341867s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458231853 (8h14m12.396647702s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458325132 (8h14m12.396740981s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458575431 (8h14m12.39699128s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458624104 (8h14m12.397039953s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458906444 (8h14m12.397322293s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.458920901 (8h14m12.39733676s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459170670 (8h14m12.397586519s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459279838 (8h14m12.397695687s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459426770 (8h14m12.397842619s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459604800 (8h14m12.398020639s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.459897921 (8h14m12.39831377s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:04.460116189 (8h14m12.398532038s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Uploading piece... Aug 6 22:23:06.263153507 (8h14m14.201569346s) [upload=7849]: Next returned nil segment Aug 6 22:23:06.263169687 (8h14m14.201585536s) [upload=7849]: Waiting for error group managing segments... Aug 6 22:23:06.593041172 (8h14m14.531457011s) [noise=true , address=77.90.31.17:4771 , piece_id=W6RPZPLNICF5OJ57IOCOSFEPA2ROLGJLCGS3WHJ3AIEYKIJNWQNQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.614464201 (8h14m14.55288005s) [noise=true , address=77.90.31.19:4841 , piece_id=ZBSVBU6WFTKXASVXXYWT4GT3MSOP5WMCQBUV6YE5AOYCCEJ5MPIQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.646281806 (8h14m14.584697655s) [noise=true , address=45.91.116.9:4261 , piece_id=ETGKKVWZIQKRHE7VOYK7YYNAVVLM5J4SAZZKHDNXNGCRV7NTMG5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.684232226 (8h14m14.622648075s) [noise=true , address=104.36.82.11:4113 , piece_id=HGUZT3RVVLFQJKABNOMMZR76OWJ5UUQNWNTP5XIZX3E5FXIUSL7A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.736100365 (8h14m14.674516214s) [noise=true , address=104.36.82.5:4838 , piece_id=SUPJYU4EXQW3NGJKFSZVVGD2LFIA44SCWMMUZIO35XYFGQRWWLAQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.744475628 (8h14m14.682891477s) [noise=true , address=77.90.31.21:4878 , piece_id=CJOZY23NGALJWBBKWS57YVSSPCX3LRP255HGWD7MNRK7J2XDTC5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.775567917 (8h14m14.713983766s) [noise=true , address=77.90.31.10:4519 , piece_id=4UO7DK73UJS5BS7NKRIAJQTUNUJSV5Q2I4K53DRC6XSEBCGERNSQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.785843175 (8h14m14.724259025s) [noise=true , address=45.91.116.14:4190 , piece_id=ZE6HGQG2H3JJQYK4BHNY3LTWGV7LMVFLHPIFUAXSZZP723PQRFZA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.793127933 (8h14m14.731543773s) [noise=true , address=45.91.116.12:4227 , piece_id=J42ZCYMEV5FIA2Q2DV2JDHWSFLK4D2B7XYGGW2X7S3T6XKRIORDA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.805200522 (8h14m14.743616371s) [noise=true , address=104.36.82.7:4912 , piece_id=T55YOIIG3W2KVSGZDLVGHHIW2J4ONW7XR6TZ5SDOM2KSZBG6P4QA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.840123841 (8h14m14.77853969s) [noise=true , address=45.91.116.11:4117 , piece_id=S4B7BROXVTGZPQSHUDA3P5OBFRKHFJXVRFTULNKLGBNZ7VLIA2SQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.895304316 (8h14m14.833720165s) [noise=true , address=45.91.116.4:4938 , piece_id=M6TYOYGHNPPOY7WPUDPRGK67IN3V6SNR54DGM4GH5OHE7HFGMUTA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915155747 (8h14m14.853571596s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZZBTCWYIOUN7772Y5NDMLIOBWZRRPS3APLXEO7CNVDG5GBE2E5Q, seg_pos={0 0}, upload=7849]: Done uploading piece. err: <nil> Aug 6 22:23:06.915179071 (8h14m14.85359492s) [seg_pos={0 0}, upload=7849]: Segment reached optimal threshold of 13 pieces. Aug 6 22:23:06.915296927 (8h14m14.853712816s) [noise=true , address=77.90.31.18:4805 , piece_id=6EOOOZDJB2XLT53M3FZ3KNWNVVJANMXRTBNXDATV373CLDEXAN5A, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 6 22:23:06.915493824 (8h14m14.853909673s) [noise=true , address=77.90.31.13:4626 , piece_id=VPJSTULWVUCWYJ252H47BXUSBLAUCXZJUYOOK7X6RXTKTV3HP3TQ, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43736->77.90.31.13:4626: use of closed network connection Aug 6 22:23:06.915533600 (8h14m14.853949449s) [noise=true , address=104.36.82.3:4767 , piece_id=I4S5BOGIOLZAOZL7HWBN3BO6KDQTKDKYLJAQUP3FBL4ICBI4ILSA, seg_pos={0 0}, upload=7849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: read tcp 188.166.225.170:33788->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 6 22:23:06.915551634 (8h14m14.853967483s) [seg_pos={0 0}, upload=7849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 6 22:23:06.915610066 (8h14m14.854025915s) [upload=7849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 6 22:23:06.915613933 (8h14m14.854029772s) [upload=7849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 6 22:23:07.212500018 (8h14m15.150915887s) [upload=7849]: Done uploading segments... Aug 6 22:23:07.212522912 (8h14m15.150938751s) [upload=7849]: Upload finished. err: <nil> Duration: 30.00194955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.............X............................] duration: 7.312264667s FAIL (2025-08-06 22:23:26 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 7.312289124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.969376359s FAIL (2025-08-06 22:22:53 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.969422217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............X...........................] duration: 8.860373094s FAIL (2025-08-06 22:22:13 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 8.860399664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.222107091s FAIL (2025-08-06 22:21:41 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.22212629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............X..........................] duration: 9.302345767s FAIL (2025-08-06 22:21:21 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 9.302373661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............X.........................] duration: 6.197679719s FAIL (2025-08-06 22:20:52 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.197702552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.868920418s FAIL (2025-08-06 22:20:41 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.86895352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.157495143s FAIL (2025-08-06 22:20:05 UTC) orbiter warning: operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.157524429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............X........................] duration: 6.848364619s FAIL (2025-08-06 22:19:57 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 6.84838656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............X.......................] duration: 9.675093308s FAIL (2025-08-06 22:19:34 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 797b4dc7c9336de14c0d9edcd4fc93fa494b55e0fda60d4b63baf4e5feb284a3 != before bf3c086bffc64c0cfe40b021cb4adf18ff1d13140d4640a9fd1d783e23a29b11 Duration: 9.675117474s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)