Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X...X..................X.....................] duration: 1.843012397s FAIL (2025-08-11 04:27:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.843038398s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..............X...........................] duration: 7.642892259s FAIL (2025-08-11 04:26:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.642918099s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.956789662s FAIL (2025-08-11 04:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.956815221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X...X..................X....................] duration: 1.829912973s FAIL (2025-08-11 04:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.829938633s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 8.307390939s FAIL (2025-08-11 04:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.307414154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..............X..........................] duration: 7.687268532s FAIL (2025-08-11 04:25:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.687295304s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X...X..................X...................] duration: 1.833774621s FAIL (2025-08-11 04:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.833804419s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..............X.........................] duration: 7.328324704s FAIL (2025-08-11 04:25:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.328350002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 9.175461053s FAIL (2025-08-11 04:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.175493435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X...X..................X..................] duration: 1.839755748s FAIL (2025-08-11 04:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.839784293s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..............X........................] duration: 8.622197109s FAIL (2025-08-11 04:24:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.622227819s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..............X.......................] duration: 8.929998279s FAIL (2025-08-11 04:24:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.93002462s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X..................X...X..................X.................] duration: 1.843892291s FAIL (2025-08-11 04:24:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.843918611s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.448129945s FAIL (2025-08-11 04:23:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.4481581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X..................X...X..................X................] duration: 1.891964879s FAIL (2025-08-11 04:23:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.891994516s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..............X......................] duration: 7.852844511s FAIL (2025-08-11 04:23:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.852869107s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.178406913s FAIL (2025-08-11 04:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.178431671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X..................X...X..................X...............] duration: 1.977237111s FAIL (2025-08-11 04:23:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.977258112s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..............X.....................] duration: 7.717944061s FAIL (2025-08-11 04:23:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.717971343s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..............X....................] duration: 7.999319473s FAIL (2025-08-11 04:22:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.999340233s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...X..................X...X..................X..............] duration: 1.819848379s FAIL (2025-08-11 04:22:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.819885118s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.753654611s FAIL (2025-08-11 04:22:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.753681723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X........................X..............X...................] duration: 8.865048863s FAIL (2025-08-11 04:22:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.865075654s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X..................X...X..................X.............] duration: 1.876375991s FAIL (2025-08-11 04:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.876403494s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.847126636s FAIL (2025-08-11 04:21:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.847172253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 8.562743565s FAIL (2025-08-11 04:21:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.562773772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X........................X..............X..................] duration: 7.653305867s FAIL (2025-08-11 04:21:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.653340644s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X..................X...X..................X............] duration: 1.82047742s FAIL (2025-08-11 04:20:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.820512157s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X..................X...X..................X...........] duration: 1.979769645s FAIL (2025-08-11 04:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.979802428s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 9.354010065s FAIL (2025-08-11 04:20:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.354036055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X........................X..............X.................] duration: 9.499502439s FAIL (2025-08-11 04:20:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 9.499529118s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X........................X..............X................] duration: 9.597571899s FAIL (2025-08-11 04:20:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 9.597602267s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X..................X...X..................X..........] duration: 1.811071284s FAIL (2025-08-11 04:20:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.811108044s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.877410834s FAIL (2025-08-11 04:19:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.877467933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X........................X..............X...............] duration: 7.831796318s FAIL (2025-08-11 04:19:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.83183445s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........X..................X...X..................X.........] duration: 1.988465644s FAIL (2025-08-11 04:19:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.988603747s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.278608593s FAIL (2025-08-11 04:19:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.278635875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 10.313266865s FAIL (2025-08-11 04:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.313286914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...X........................X..............X..............] duration: 8.436748621s FAIL (2025-08-11 04:18:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.436793426s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.805312501s FAIL (2025-08-11 04:18:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.805339782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........X..................X...X..................X........] duration: 1.826556462s FAIL (2025-08-11 04:18:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.826584466s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X........................X..............X.............] duration: 8.490493983s FAIL (2025-08-11 04:18:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.490514342s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.191742979s FAIL (2025-08-11 04:17:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.19176924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........X..................X...X..................X.......] duration: 2.004770893s FAIL (2025-08-11 04:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 2.00479573s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X...X........................X..............X............] duration: 8.422155924s FAIL (2025-08-11 04:17:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 8.422181334s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...X........................X..............X...........] duration: 7.973136509s FAIL (2025-08-11 04:17:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.973164963s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.771703534s FAIL (2025-08-11 04:16:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.771731067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........X..................X...X..................X......] duration: 1.869160858s FAIL (2025-08-11 04:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f84b1d63f143b6fa4eec3e432b35e437fdb8d3fdbb445f4133a084369fe043dc != before 27d66f54be915b8e04be0f96f098f1c6f16228a094324d3e443337a9745c0d32 ======LOGS====== Aug 11 04:11:08.678263761 (58h45m3.109343882s) [upload=61229]: Starting upload Aug 11 04:11:08.678315781 (58h45m3.109395912s) [upload=61229]: Done starting upload Aug 11 04:11:08.678370285 (58h45m3.109450416s) [upload=61229]: Uploading segments... Aug 11 04:11:08.678406083 (58h45m3.109486214s) [upload=61229]: Got next segment. Inline: true Aug 11 04:11:08.678410010 (58h45m3.109490141s) [upload=61229]: Waiting for error group managing segments... Aug 11 04:11:08.678442923 (58h45m3.109523054s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:11:08.678449116 (58h45m3.109529247s) [upload=61229]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:11:08.678451400 (58h45m3.109531531s) [upload=61229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:11:08.981325999 (58h45m3.41240613s) [upload=61229]: Done uploading segments... Aug 11 04:11:08.981343683 (58h45m3.412423814s) [upload=61229]: Upload finished. err: <nil> Duration:* 1.869192007s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.298295608s FAIL (2025-08-11 04:16:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.298321359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...X........................X..............X..........] duration: 7.397632427s FAIL (2025-08-11 04:16:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:10:12.024128950 (58h44m6.455209081s) [upload=61212]: Starting upload Aug 11 04:10:12.024174448 (58h44m6.455254579s) [upload=61212]: Done starting upload Aug 11 04:10:12.024310668 (58h44m6.455390799s) [upload=61212]: Uploading segments... Aug 11 04:10:12.024382615 (58h44m6.455462756s) [upload=61212]: Got next segment. Inline: false Aug 11 04:10:12.024418424 (58h44m6.455498556s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:10:12.024424105 (58h44m6.455504236s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:10:12.470212275 (58h44m6.901292406s) [seg_pos={0 0}, upload=61212]: Begin upload segment... Aug 11 04:10:12.470263423 (58h44m6.901343554s) [seg_pos={0 0}, upload=61212]: Done begin upload segment. Aug 11 04:10:12.470286758 (58h44m6.901366889s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470327797 (58h44m6.901407928s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470643430 (58h44m6.901723561s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470751307 (58h44m6.901831438s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.470907305 (58h44m6.901987436s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471065068 (58h44m6.902145199s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471180739 (58h44m6.90226087s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471423974 (58h44m6.902504105s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471677739 (58h44m6.902757871s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471693208 (58h44m6.90277337s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.471912057 (58h44m6.902992178s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472204086 (58h44m6.903284227s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472473352 (58h44m6.903553483s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472664868 (58h44m6.903744999s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.472857657 (58h44m6.903937788s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473061437 (58h44m6.904141568s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473260558 (58h44m6.904340679s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:12.473448828 (58h44m6.904528959s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Uploading piece... Aug 11 04:10:13.655946677 (58h44m8.087026808s) [upload=61212]: Next returned nil segment Aug 11 04:10:13.655965673 (58h44m8.087045805s) [upload=61212]: Waiting for error group managing segments... Aug 11 04:10:13.968822622 (58h44m8.399902764s) [noise=true , address=193.149.164.143:4695 , piece_id=GTGQPLRGLZMP6QMBG2R3PJBK7IMRARZ2YQYEENI3TXR5NCU2U7XA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.331595787 (58h44m8.762675948s) [noise=true , address=64.49.13.155:4923 , piece_id=D6VNV7MLPHL67ZNGEX2372NEBFXMX63WIOO2SSKOUAJZNOK3YT7A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.678621282 (58h44m9.109701413s) [noise=true , address=104.36.82.58:4299 , piece_id=AL23C5QEV6B7BPAWAR6E6YWK4EHWQT3ZYU34CMCFIOY5XDEHT3OA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.967388538 (58h44m9.398468669s) [noise=true , address=104.36.82.18:4371 , piece_id=BFC2BLUEZNT3JWTWYNDFM2N63QFAH7FFVDN7XUO6M6W2HKE4TKOQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:14.982733118 (58h44m9.413813249s) [noise=true , address=104.36.82.15:4290 , piece_id=LTJOLHC5H2R3BZUZANP44IKRVJCBS2YJBP4OE47V234CHSDQNIJA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:15.175330911 (58h44m9.606411042s) [noise=true , address=104.36.82.12:4162 , piece_id=4YJRJZ43L4JAGWYXGQYFVNHJ6JCM2SBLC4R53SUKXFE65YQTTXRQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.025565120 (58h44m10.456645301s) [noise=true , address=104.36.82.9:4035 , piece_id=5SQUGLS6B7X5EQLUMDZ6JDVYGJKHUIV672SPPJM3M4Y36CNZ5WZA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.055424641 (58h44m10.486504772s) [noise=true , address=104.36.82.46:4491 , piece_id=7DYUVFPV4JF7UUTDGUJ4H4SMXB4HTS4BFIYOQOSRXWT4U3WLBMCQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.202095150 (58h44m10.633175281s) [noise=true , address=104.36.82.16:4312 , piece_id=2UYJIQAJUC6AEQ2LUOU7F3CYG3LIKETSO5T3VD7LCMNUXX4P5ZCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.305406307 (58h44m10.736486439s) [noise=true , address=104.36.82.60:4346 , piece_id=GQ6WN4EX3TV3APHORSPB3WALSW7OQH372H6WLCLBNI7N44DSQXFA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.343736136 (58h44m10.774816267s) [noise=true , address=104.36.82.47:4544 , piece_id=NIHQXDI4KWMAUGEULZFAUC2XHCUQP4YIBFNEKIRTX3LWA4SUA4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.345910396 (58h44m10.776990527s) [noise=true , address=77.90.31.19:4874 , piece_id=BHM7DCYX74GW6G32AFBI6CIAZPD5TOEBU5IB2T4AWWPYWXNBRTWQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.374390036 (58h44m10.805470167s) [noise=true , address=104.36.82.13:4210 , piece_id=J3CWGH3V4DF3ZMFIWWZZ5XT5UVC5P4CJU6DLQ7LJA6B27NY72JCA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573505951 (58h44m11.004586082s) [noise=true , address=104.36.82.44:4876 , piece_id=HFEQFC5GCJ646YSWPYZUY3N3TJZH2G3CRNKD3ULHWOW5UPRRSOJQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: <nil> Aug 11 04:10:16.573536640 (58h44m11.004616771s) [seg_pos={0 0}, upload=61212]: Segment reached optimal threshold of 14 pieces. Aug 11 04:10:16.573703930 (58h44m11.004784061s) [noise=true , address=104.36.82.23:4473 , piece_id=O25NAOPOUQNKAG4DZMILAPZHN2NGZ36ICD5ANZQVJYREYX6JC4HA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573719550 (58h44m11.004799671s) [noise=true , address=104.36.82.39:4742 , piece_id=XH3YH4XLUEHVMWI4A6S4RRRI2FDKSUCXTOSHVCBNTLKOS7FSJJLQ, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573783081 (58h44m11.004863202s) [noise=true , address=104.36.82.34:4545 , piece_id=XPPKM5NYMCD5SA5V5X6YCGPFGM7AEO2UXDDGALQDHTSB7OSLCJQA, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34182->104.36.82.34:4545: use of closed network connection Aug 11 04:10:16.573912860 (58h44m11.004992991s) [noise=true , address=104.36.82.19:4408 , piece_id=H3TMKFQTXSXKQ4ZUVRGFYLPUWA5B2PIJ6JD2WJVY2RDC72KRT57A, seg_pos={0 0}, upload=61212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:10:16.573927919 (58h44m11.00500805s) [seg_pos={0 0}, upload=61212]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:10:16.574032939 (58h44m11.00511307s) [upload=61212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:10:16.574037098 (58h44m11.005117229s) [upload=61212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:10:16.871529179 (58h44m11.30260931s) [upload=61212]: Done uploading segments... Aug 11 04:10:16.871553165 (58h44m11.302633296s) [upload=61212]: Upload finished. err: <nil> Duration: 7.397657406s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)