Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.234992597s FAIL (2025-08-16 15:35:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== VVSG756LMZQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.867897576 (190h8m28.298977707s) [noise=true , address=104.36.82.35:4599 , piece_id=O2ZPAFMGZ5LVXEZP5MLMEFJ4MQWKYB3J26FNNLHHQYTHQQWTOSPQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.868169847 (190h8m28.299249988s) [noise=true , address=104.36.82.17:4349 , piece_id=YRQVKWLW7VERWX46F5RHYPGCPOCBZTNMKFJCMQCPLKEAZEUJIIWQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.868485481 (190h8m28.299565612s) [noise=true , address=104.36.82.12:4162 , piece_id=4FXY2EP7VNL2YMSYLLG6FDZNQLGBCFZDP3C3ZBMTEYSYCYUAQZ7Q, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.868709930 (190h8m28.299790061s) [noise=true , address=79.127.201.215:4353 , piece_id=QEG5BVPN7RXREJPGCRBWWN3RUPWB473P4FF4FY3FTXLNT6VPXNHQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.868937616 (190h8m28.300017747s) [noise=true , address=104.36.82.5:4855 , piece_id=27I2IELBXDO26JKEQFFEJYEXVJRJHILJYGOV4H4NUFY27JNCSYDA, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.869133922 (190h8m28.300214043s) [noise=true , address=45.91.116.5:4326 , piece_id=SJSN2OG5JXIGTMIJNT6XBR3SKRN7BNTWDH4Q77FEB7X72S7VNHUQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.869150694 (190h8m28.300230825s) [noise=true , address=66.23.193.175:4554 , piece_id=BIHMUGOETJZBX5U6PUKPOEFGNTPTXTRZMSKK5K55BKBZ7DV37SEQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.869192734 (190h8m28.300272855s) [noise=true , address=77.90.31.16:4745 , piece_id=EUWKH3OJRSJT34XJTBDGE3CDGDOWA7BCD22O4VRXCDG3UZ6TRZ5Q, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.869528055 (190h8m28.300608186s) [noise=true , address=104.36.82.38:4718 , piece_id=WSIJY5PIX6PM2RBPUIHI4OFPLHZSKZVVFDCCBBQ7BUXCKZ4GSJTQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.869865392 (190h8m28.300945523s) [noise=true , address=104.36.82.60:4342 , piece_id=6K4KSECAUKN4DPY4YF75NH2TNUBXYCCKV6CXQ6U6FZMXPCOTR3SA, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.872838961 (190h8m28.303919092s) [noise=true , address=104.36.82.58:4265 , piece_id=3R3UTPTK3OUAVF5PRCCEUUXPOUVMN46FB5ILZXJUMY2BWX3AC5DQ, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.874021714 (190h8m28.305101845s) [noise=true , address=77.90.31.12:4610 , piece_id=VKMK2TUKZ25YVZTK22RWKSDEEBFEPNQXJYKYQYHK43AYC6MJ3KHA, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.874454292 (190h8m28.305534423s) [noise=true , address=104.36.82.18:4378 , piece_id=PJF4WRJ3BKJS2N76VRORFBJGVGS4EXSYSEJKYPEWFC4ACSKI467A, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:33.874926645 (190h8m28.306006766s) [noise=true , address=104.36.82.2:4759 , piece_id=2WFGELSSTPXSDKIIGRAUXOB4G5UUK7HRDJ3D5W3T3BQUMAEAP5SA, seg_pos={0 0}, upload=197820]: Uploading piece... Aug 16 15:34:35.030569050 (190h8m29.461649181s) [upload=197820]: Next returned nil segment Aug 16 15:34:35.030585302 (190h8m29.461665433s) [upload=197820]: Waiting for error group managing segments... Aug 16 15:34:35.996100886 (190h8m30.427181017s) [noise=true , address=104.36.82.12:4162 , piece_id=4FXY2EP7VNL2YMSYLLG6FDZNQLGBCFZDP3C3ZBMTEYSYCYUAQZ7Q, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:36.479968302 (190h8m30.911048433s) [noise=true , address=77.90.31.14:4675 , piece_id=V2ODDXALUZN2WZL2MR4EY7I6LLQIXHGUSGGBX3TURVVSG756LMZQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:36.655256510 (190h8m31.086336641s) [noise=true , address=77.90.31.16:4745 , piece_id=EUWKH3OJRSJT34XJTBDGE3CDGDOWA7BCD22O4VRXCDG3UZ6TRZ5Q, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:36.725055424 (190h8m31.156135555s) [noise=true , address=77.90.31.12:4610 , piece_id=VKMK2TUKZ25YVZTK22RWKSDEEBFEPNQXJYKYQYHK43AYC6MJ3KHA, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:37.062042337 (190h8m31.493122468s) [noise=true , address=77.90.31.18:4826 , piece_id=G3XAJDRPYP4FDSHVCEGSGSDPIDPN4PY7FWFSO7CTVZJ6UA36LWHQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:39.538154308 (190h8m33.969234439s) [noise=true , address=45.91.116.5:4326 , piece_id=SJSN2OG5JXIGTMIJNT6XBR3SKRN7BNTWDH4Q77FEB7X72S7VNHUQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:34:42.326084235 (190h8m36.757164366s) [noise=true , address=66.23.193.175:4554 , piece_id=BIHMUGOETJZBX5U6PUKPOEFGNTPTXTRZMSKK5K55BKBZ7DV37SEQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:35:07.358171053 (190h9m1.789251184s) [noise=true , address=79.127.201.215:4353 , piece_id=QEG5BVPN7RXREJPGCRBWWN3RUPWB473P4FF4FY3FTXLNT6VPXNHQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: <nil> Aug 16 15:35:32.769104447 (190h9m27.200184579s) [noise=true , address=104.36.82.35:4599 , piece_id=O2ZPAFMGZ5LVXEZP5MLMEFJ4MQWKYB3J26FNNLHHQYTHQQWTOSPQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:1nxKQjpzyhZsRnkffX1fwCXcTYiwHPwuNg4HWtpdzRnKQCtp6s, address:104.36.82.35:4599): protocol: rpc: dial tcp 104.36.82.35:4599: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769156136 (190h9m27.200236267s) [noise=true , address=104.36.82.38:4718 , piece_id=WSIJY5PIX6PM2RBPUIHI4OFPLHZSKZVVFDCCBBQ7BUXCKZ4GSJTQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98, address:104.36.82.38:4718): protocol: rpc: dial tcp 104.36.82.38:4718: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769187126 (190h9m27.200267247s) [noise=true , address=104.36.82.60:4342 , piece_id=6K4KSECAUKN4DPY4YF75NH2TNUBXYCCKV6CXQ6U6FZMXPCOTR3SA, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:104.36.82.60:4342): protocol: rpc: dial tcp 104.36.82.60:4342: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769303749 (190h9m27.20038388s) [noise=true , address=104.36.82.58:4265 , piece_id=3R3UTPTK3OUAVF5PRCCEUUXPOUVMN46FB5ILZXJUMY2BWX3AC5DQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:104.36.82.58:4265): protocol: write tcp 188.166.225.170:32856->104.36.82.58:4265: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769360287 (190h9m27.200440418s) [noise=true , address=104.36.82.44:4880 , piece_id=AKYJQIVFVS4VJHVORFMRTD5UAQGWQ3FOHEVSFPW24BWH4WIHSRIQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: write tcp 188.166.225.170:42562->104.36.82.44:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769407829 (190h9m27.20048796s) [noise=true , address=104.36.82.17:4349 , piece_id=YRQVKWLW7VERWX46F5RHYPGCPOCBZTNMKFJCMQCPLKEAZEUJIIWQ, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:1zywsXJnw5gfpUu7mC8p7UE7ev5HexPu5nSpUH1x2pzar99T97, address:104.36.82.17:4349): protocol: write tcp 188.166.225.170:50598->104.36.82.17:4349: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769443967 (190h9m27.200524098s) [noise=true , address=104.36.82.2:4759 , piece_id=2WFGELSSTPXSDKIIGRAUXOB4G5UUK7HRDJ3D5W3T3BQUMAEAP5SA, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: write tcp 188.166.225.170:54324->104.36.82.2:4759: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769471751 (190h9m27.200551872s) [noise=true , address=104.36.82.59:4333 , piece_id=4CF6QCTKA4ATJV6S4QLOXV5XUVVHKWZJFD74CIOE2UMLHQBY5YOA, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN, address:104.36.82.59:4333): protocol: write tcp 188.166.225.170:40270->104.36.82.59:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769486288 (190h9m27.200566419s) [noise=true , address=104.36.82.5:4855 , piece_id=27I2IELBXDO26JKEQFFEJYEXVJRJHILJYGOV4H4NUFY27JNCSYDA, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:33194->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769518871 (190h9m27.200599002s) [noise=true , address=104.36.82.18:4378 , piece_id=PJF4WRJ3BKJS2N76VRORFBJGVGS4EXSYSEJKYPEWFC4ACSKI467A, seg_pos={0 0}, upload=197820]: Done uploading piece. err: ecclient: upload failed (node:13NZR1Ah43qF6pvFdUpwSrJiCx1HZf3Rb1zwRaPs6tv6inUaQi, address:104.36.82.18:4378): protocol: write tcp 188.166.225.170:44036->104.36.82.18:4378: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:32.769557174 (190h9m27.200637305s) [seg_pos={0 0}, upload=197820]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:35:33.002116835 (190h9m27.433196966s) [upload=197820]: Done uploading segments... Aug 16 15:35:33.002141042 (190h9m27.433221173s) [upload=197820]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.235012947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [............X...X...X..................X.X..X.....X.........] duration: 4.559422655s FAIL (2025-08-16 15:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 4.559448272s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................X......X..XXXX] duration: 30.23513045s FAIL (2025-08-16 15:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:34:26.504644080 (190h8m20.935724211s) [upload=197818]: Starting upload Aug 16 15:34:26.504699557 (190h8m20.935779688s) [upload=197818]: Done starting upload Aug 16 15:34:26.504794568 (190h8m20.935874699s) [upload=197818]: Uploading segments... Aug 16 15:34:26.504898156 (190h8m20.935978287s) [upload=197818]: Got next segment. Inline: false Aug 16 15:34:26.504927763 (190h8m20.936007894s) [upload=197818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.504932412 (190h8m20.936012533s) [upload=197818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.738022307 (190h8m21.169102428s) [seg_pos={0 0}, upload=197818]: Begin upload segment... Aug 16 15:34:26.738058327 (190h8m21.169138448s) [seg_pos={0 0}, upload=197818]: Done begin upload segment. Aug 16 15:34:26.738078415 (190h8m21.169158546s) [noise=true , address=104.36.82.8:4971 , piece_id=FPW2C54A3OSDZQIC364FOV3LLSSUPTZ4PYVF5JWAVILXYGVPA6PA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.738499210 (190h8m21.169579342s) [noise=true , address=104.36.82.4:4804 , piece_id=G2MLQALAZ7NIRQK6W6OQRM6GJQS6LODO2RCBW2XINDKC7GBWBCFA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.738508849 (190h8m21.16958898s) [noise=true , address=104.36.82.10:4078 , piece_id=NVCOASQ6G3SY27UVWRIUUCWE62HT3GBNHCXDYMRSB72OARWPLKIQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.738793413 (190h8m21.169873544s) [noise=true , address=104.36.82.13:4187 , piece_id=7MBD5NHYPAR6TMEZOMKHABB7YN2JY5CKJ56T5BUTUKXFSOSUPZKQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739066686 (190h8m21.170146817s) [noise=true , address=104.36.82.2:4729 , piece_id=O6QS3236G5KFHWW4VL6KJMPAORVYFHU3EAE4WAKXPYT2MHFK3ADA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739289122 (190h8m21.170369253s) [noise=true , address=104.36.82.11:4115 , piece_id=5O2WJNENSO2AQNTKCRWLWQ4XXUTWK5FHKIGTWP2HXXK2IXR45TPQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739483644 (190h8m21.170563775s) [noise=true , address=77.90.31.12:4589 , piece_id=7R4KQ35IN7YQISPH7VHM4PFRPZVUWASQGIFBETS2UYGJEWVMV5IQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739537567 (190h8m21.170617698s) [noise=true , address=45.91.116.14:4189 , piece_id=6GIL6UJFQENF2K4EBHT2MTYXEDUTHOBIWYKXD7PRGHUETGUKTANA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739738812 (190h8m21.170818943s) [noise=true , address=45.91.116.11:4118 , piece_id=PGC2KCDOB4WYRMW2R5RH34CWIUHT342YW6557EKADFQC6KLNOJJQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739772406 (190h8m21.170852537s) [noise=true , address=104.36.82.15:4258 , piece_id=SDDIWIVJNX6UDWVERKHM6JNLULTX3OZTQPLBLNRW7WBV5HRYHFBQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739979964 (190h8m21.171060085s) [noise=true , address=104.36.82.7:4912 , piece_id=CV6N45RCZA56D22RYJ4JN45LKFKKPATE34FFXNQFQ2TCJU3YLE5Q, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.739995703 (190h8m21.171075834s) [noise=true , address=45.91.116.11:4117 , piece_id=ZQ4ST2MWQ4QYDIOEY7MZ7Q3LEUDNBLUQ76BVJAPIW5RWUJMQNXJA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.740230453 (190h8m21.171310584s) [noise=true , address=77.90.31.12:4590 , piece_id=XX7E3HL7ZTIGVYOTW6RJMTCDHRIZW6VSJDNQDUUZPL3VHCSFMZQA, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.740239119 (190h8m21.17131925s) [noise=true , address=104.36.82.14:4222 , piece_id=QWXSC65PWMJ3XVCDDTME4HW2BNRWXUHPJGQIM5FTKURRY7UOQQVQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.740518844 (190h8m21.171598975s) [noise=true , address=104.36.82.5:4840 , piece_id=PL76O3HHE3T3H2F6V2TBYK3UN6WF3GSUWVW3CBXP2A3BDNKN2ANQ, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:26.744962748 (190h8m21.176042879s) [noise=true , address=104.36.82.6:4875 , piece_id=4FRT2DUMHYUDUQO3FQYERBZMX44TQSXCGCJGWNDVWKNUEBAFSP4Q, seg_pos={0 0}, upload=197818]: Uploading piece... Aug 16 15:34:28.493567532 (190h8m22.924647663s) [upload=197818]: Next returned nil segment Aug 16 15:34:28.493587701 (190h8m22.924667832s) [upload=197818]: Waiting for error group managing segments... Aug 16 15:34:28.782760246 (190h8m23.213840377s) [noise=true , address=104.36.82.10:4078 , piece_id=NVCOASQ6G3SY27UVWRIUUCWE62HT3GBNHCXDYMRSB72OARWPLKIQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:28.795331439 (190h8m23.22641157s) [noise=true , address=104.36.82.15:4258 , piece_id=SDDIWIVJNX6UDWVERKHM6JNLULTX3OZTQPLBLNRW7WBV5HRYHFBQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:28.849666214 (190h8m23.280746345s) [noise=true , address=104.36.82.7:4912 , piece_id=CV6N45RCZA56D22RYJ4JN45LKFKKPATE34FFXNQFQ2TCJU3YLE5Q, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:28.946023685 (190h8m23.377103816s) [noise=true , address=104.36.82.6:4875 , piece_id=4FRT2DUMHYUDUQO3FQYERBZMX44TQSXCGCJGWNDVWKNUEBAFSP4Q, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:28.975746787 (190h8m23.406826918s) [noise=true , address=104.36.82.11:4115 , piece_id=5O2WJNENSO2AQNTKCRWLWQ4XXUTWK5FHKIGTWP2HXXK2IXR45TPQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:30.699872492 (190h8m25.130952623s) [noise=true , address=77.90.31.12:4590 , piece_id=XX7E3HL7ZTIGVYOTW6RJMTCDHRIZW6VSJDNQDUUZPL3VHCSFMZQA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:30.719564943 (190h8m25.150645074s) [noise=true , address=45.91.116.14:4189 , piece_id=6GIL6UJFQENF2K4EBHT2MTYXEDUTHOBIWYKXD7PRGHUETGUKTANA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:30.897487271 (190h8m25.328567402s) [noise=true , address=77.90.31.12:4589 , piece_id=7R4KQ35IN7YQISPH7VHM4PFRPZVUWASQGIFBETS2UYGJEWVMV5IQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:31.603932598 (190h8m26.035012759s) [noise=true , address=45.91.116.11:4118 , piece_id=PGC2KCDOB4WYRMW2R5RH34CWIUHT342YW6557EKADFQC6KLNOJJQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:35.980122142 (190h8m30.411202273s) [noise=true , address=45.91.116.11:4117 , piece_id=ZQ4ST2MWQ4QYDIOEY7MZ7Q3LEUDNBLUQ76BVJAPIW5RWUJMQNXJA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: <nil> Aug 16 15:34:55.854472978 (190h8m50.285553109s) [noise=true , address=104.36.82.5:4840 , piece_id=PL76O3HHE3T3H2F6V2TBYK3UN6WF3GSUWVW3CBXP2A3BDNKN2ANQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854522943 (190h8m50.285603064s) [noise=true , address=104.36.82.14:4222 , piece_id=QWXSC65PWMJ3XVCDDTME4HW2BNRWXUHPJGQIM5FTKURRY7UOQQVQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:39572->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854553832 (190h8m50.285633963s) [noise=true , address=104.36.82.8:4971 , piece_id=FPW2C54A3OSDZQIC364FOV3LLSSUPTZ4PYVF5JWAVILXYGVPA6PA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:58046->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854612505 (190h8m50.285692636s) [noise=true , address=104.36.82.2:4729 , piece_id=O6QS3236G5KFHWW4VL6KJMPAORVYFHU3EAE4WAKXPYT2MHFK3ADA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854641069 (190h8m50.2857212s) [noise=true , address=104.36.82.4:4804 , piece_id=G2MLQALAZ7NIRQK6W6OQRM6GJQS6LODO2RCBW2XINDKC7GBWBCFA, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854666618 (190h8m50.285746749s) [noise=true , address=104.36.82.13:4187 , piece_id=7MBD5NHYPAR6TMEZOMKHABB7YN2JY5CKJ56T5BUTUKXFSOSUPZKQ, seg_pos={0 0}, upload=197818]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:55.854694752 (190h8m50.285774883s) [seg_pos={0 0}, upload=197818]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:34:56.087753538 (190h8m50.518833669s) [upload=197818]: Done uploading segments... Aug 16 15:34:56.087777695 (190h8m50.518857826s) [upload=197818]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235150868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X...........X......X...X.X......X..X......X..X...XXXXXX] duration: 1m0.249693806s FAIL (2025-08-16 15:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:33:37.856053143 (190h7m32.287133274s) [upload=197807]: Starting upload Aug 16 15:33:37.856094552 (190h7m32.287174683s) [upload=197807]: Done starting upload Aug 16 15:33:37.856205414 (190h7m32.287285545s) [upload=197807]: Uploading segments... Aug 16 15:33:37.856356023 (190h7m32.287436174s) [upload=197807]: Got next segment. Inline: false Aug 16 15:33:37.856424854 (190h7m32.287505025s) [upload=197807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:37.856432049 (190h7m32.28751218s) [upload=197807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:38.362316011 (190h7m32.793396172s) [seg_pos={0 0}, upload=197807]: Begin upload segment... Aug 16 15:33:38.362365084 (190h7m32.793445245s) [seg_pos={0 0}, upload=197807]: Done begin upload segment. Aug 16 15:33:38.362814735 (190h7m32.793894876s) [noise=true , address=104.36.82.4:4835 , piece_id=MGYXYRMNSHYATOLPIB6CAVJI4CKPDLMVK2MRYTH53P6VWU2M4CSA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.363817693 (190h7m32.794897835s) [noise=true , address=104.36.82.9:4025 , piece_id=QYFK6UEJOTBWIIFQHTMW76JBROQV76USR362TIKQVZ5XA2SRQF4Q, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.364244991 (190h7m32.795325133s) [noise=true , address=104.36.82.38:4695 , piece_id=OGMRAXBJ24J7AUIUVI2XGICH5Q6ZFTKJEDEXR4SWU2BAKG4WSFHA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.364795545 (190h7m32.795875676s) [noise=true , address=104.36.82.37:4669 , piece_id=DHHJ3MJFSYUWUCE3XECLPZ7LQZMBK7TEV32SOYNYRJPD3PLQL2OA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.365193657 (190h7m32.796273788s) [noise=true , address=77.90.31.19:4855 , piece_id=QQZIGJEPHJD4AHU7AYA5VM4YAWYHFFPJZMFKCM5MZXIXP73S6IKQ, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.365787974 (190h7m32.796868105s) [noise=true , address=104.36.82.60:4347 , piece_id=7X37Q3T2RLG6UVXIXQHMIIGMEWQ2HGWVRI42FBBFJXGNAZGMMI3A, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.366537037 (190h7m32.797617169s) [noise=true , address=104.36.82.3:4780 , piece_id=UJWIR72LL6I2C66KKQ5EDE6SAIJX26ULOU6IM4JTFCD5JLNVZYVA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.367489780 (190h7m32.798569911s) [noise=true , address=104.36.82.5:4858 , piece_id=ZHXHXRAMD2J2R3ZVBX5SSVAMHMKT3N2JJUE5XNDIGWGT44JIH72A, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.368262850 (190h7m32.799342991s) [noise=true , address=77.90.31.18:4813 , piece_id=6EBHQTJWVCYZHP4BZZD24YA6V2LJASCTG5YN2TXG7YYRHN7PHBHA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.368623730 (190h7m32.799703861s) [noise=true , address=104.36.82.23:4475 , piece_id=FQPGUO5FPUNRO3Z4ZP4HVA4IXIMR6LHJC3U2ZHNPDPB2FFPNXPZA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.369601140 (190h7m32.800681281s) [noise=true , address=79.127.220.101:4070 , piece_id=ZBYINBUJOGLZ4TCZ2J6EWGKINR2YGHZ5UVDDLM5YSA4DSKJMXDLA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.369966319 (190h7m32.80104645s) [noise=true , address=104.36.82.58:4273 , piece_id=P2N7ABKDCS5GCPI6BBZNNMQYBYQIZFHXNGS2GWWSXFERFAWGLVCA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.370292353 (190h7m32.801372484s) [noise=true , address=45.91.116.9:4294 , piece_id=EK4HWHTK5MROYC7IASNBXYRPD4PR3FW5VLXPG3NPSX5G4LP5W23A, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.370624088 (190h7m32.801704209s) [noise=true , address=104.36.82.39:4735 , piece_id=ZP6NCLMXCBLTAQBIR6WNRBRPUNHMLK5YGA3ZLJ2S7HNPCO3NTVKQ, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.370979187 (190h7m32.802059328s) [noise=true , address=104.36.82.44:4896 , piece_id=EA43STVKAF7EFH64JHLKHX6TUBWDQAALEGXZMUJAAPCWUPEZUHCQ, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.372125930 (190h7m32.803206062s) [noise=true , address=66.23.193.175:4557 , piece_id=BGB3NXIDAPZAS6ZMC4LZ5MQYEE55G2WSPQFEN5VTGMIHCPKTBMTA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.372616730 (190h7m32.803696861s) [noise=true , address=64.49.13.155:4922 , piece_id=KGDQHANYPY4JFJCOEP5BFMZQFLCC4GTGFQYU4NAODAQ7QTCQV2OA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:38.373122929 (190h7m32.80420306s) [noise=true , address=79.127.201.215:4354 , piece_id=JPJXVMMUFJTVADEFACHP7X46RKONQXADNF5KV6XC2HMJ53Q3EGXA, seg_pos={0 0}, upload=197807]: Uploading piece... Aug 16 15:33:39.572960045 (190h7m34.004040176s) [upload=197807]: Next returned nil segment Aug 16 15:33:39.572980785 (190h7m34.004060916s) [upload=197807]: Waiting for error group managing segments... Aug 16 15:33:39.640437578 (190h7m34.071517699s) [noise=true , address=77.90.31.19:4855 , piece_id=QQZIGJEPHJD4AHU7AYA5VM4YAWYHFFPJZMFKCM5MZXIXP73S6IKQ, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: connect: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Aug 16 15:33:41.684651944 (190h7m36.115732075s) [noise=true , address=79.127.220.101:4070 , piece_id=ZBYINBUJOGLZ4TCZ2J6EWGKINR2YGHZ5UVDDLM5YSA4DSKJMXDLA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:33:44.359248928 (190h7m38.790329059s) [noise=true , address=79.127.201.215:4354 , piece_id=JPJXVMMUFJTVADEFACHP7X46RKONQXADNF5KV6XC2HMJ53Q3EGXA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:33:45.241225569 (190h7m39.6723057s) [noise=true , address=66.23.193.175:4557 , piece_id=BGB3NXIDAPZAS6ZMC4LZ5MQYEE55G2WSPQFEN5VTGMIHCPKTBMTA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:33:48.532439703 (190h7m42.963519844s) [noise=true , address=64.49.13.155:4922 , piece_id=KGDQHANYPY4JFJCOEP5BFMZQFLCC4GTGFQYU4NAODAQ7QTCQV2OA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:33:56.401944434 (190h7m50.833024565s) [noise=true , address=104.36.82.60:4347 , piece_id=7X37Q3T2RLG6UVXIXQHMIIGMEWQ2HGWVRI42FBBFJXGNAZGMMI3A, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:34:07.416464652 (190h8m1.847544813s) [noise=true , address=45.91.116.9:4294 , piece_id=EK4HWHTK5MROYC7IASNBXYRPD4PR3FW5VLXPG3NPSX5G4LP5W23A, seg_pos={0 0}, upload=197807]: Done uploading piece. err: <nil> Aug 16 15:34:37.199380762 (190h8m31.630460893s) [noise=true , address=104.36.82.5:4858 , piece_id=ZHXHXRAMD2J2R3ZVBX5SSVAMHMKT3N2JJUE5XNDIGWGT44JIH72A, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:12PasA9NPExHCQgMSK1E1kz2HHkpXQmWq7LmrwY6sRmf22avri, address:104.36.82.5:4858): protocol: write tcp 188.166.225.170:41338->104.36.82.5:4858: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199501993 (190h8m31.630582124s) [noise=true , address=104.36.82.9:4025 , piece_id=QYFK6UEJOTBWIIFQHTMW76JBROQV76USR362TIKQVZ5XA2SRQF4Q, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 188.166.225.170:57748->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199539826 (190h8m31.630619947s) [noise=true , address=77.90.31.18:4813 , piece_id=6EBHQTJWVCYZHP4BZZD24YA6V2LJASCTG5YN2TXG7YYRHN7PHBHA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:34:37.199616382 (190h8m31.630696513s) [noise=true , address=104.36.82.23:4475 , piece_id=FQPGUO5FPUNRO3Z4ZP4HVA4IXIMR6LHJC3U2ZHNPDPB2FFPNXPZA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:104.36.82.23:4475): protocol: write tcp 188.166.225.170:45822->104.36.82.23:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199650628 (190h8m31.630730759s) [noise=true , address=104.36.82.37:4669 , piece_id=DHHJ3MJFSYUWUCE3XECLPZ7LQZMBK7TEV32SOYNYRJPD3PLQL2OA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM, address:104.36.82.37:4669): protocol: write tcp 188.166.225.170:57088->104.36.82.37:4669: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199704481 (190h8m31.630784602s) [noise=true , address=104.36.82.58:4273 , piece_id=P2N7ABKDCS5GCPI6BBZNNMQYBYQIZFHXNGS2GWWSXFERFAWGLVCA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:12iN5idncnxwdBp5gdhrxbGcWi184iLJ8QA4HMk5eEEqQBCmSLt, address:104.36.82.58:4273): protocol: write tcp 188.166.225.170:59330->104.36.82.58:4273: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199742323 (190h8m31.630822444s) [noise=true , address=104.36.82.39:4735 , piece_id=ZP6NCLMXCBLTAQBIR6WNRBRPUNHMLK5YGA3ZLJ2S7HNPCO3NTVKQ, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:1QCQZ8eFGLa5TAaAJJzUip8o5DAELzTBTm4UFbt3Am5yYWXAZF, address:104.36.82.39:4735): protocol: write tcp 188.166.225.170:57746->104.36.82.39:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199770076 (190h8m31.630850197s) [noise=true , address=104.36.82.44:4896 , piece_id=EA43STVKAF7EFH64JHLKHX6TUBWDQAALEGXZMUJAAPCWUPEZUHCQ, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 188.166.225.170:39094->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199845842 (190h8m31.630925963s) [noise=true , address=104.36.82.4:4835 , piece_id=MGYXYRMNSHYATOLPIB6CAVJI4CKPDLMVK2MRYTH53P6VWU2M4CSA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:104.36.82.4:4835): protocol: write tcp 188.166.225.170:45416->104.36.82.4:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199886799 (190h8m31.63096692s) [noise=true , address=104.36.82.38:4695 , piece_id=OGMRAXBJ24J7AUIUVI2XGICH5Q6ZFTKJEDEXR4SWU2BAKG4WSFHA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y, address:104.36.82.38:4695): protocol: write tcp 188.166.225.170:50622->104.36.82.38:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199898492 (190h8m31.630978613s) [noise=true , address=104.36.82.3:4780 , piece_id=UJWIR72LL6I2C66KKQ5EDE6SAIJX26ULOU6IM4JTFCD5JLNVZYVA, seg_pos={0 0}, upload=197807]: Done uploading piece. err: ecclient: upload failed (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM, address:104.36.82.3:4780): protocol: write tcp 188.166.225.170:55666->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:37.199925143 (190h8m31.631005274s) [seg_pos={0 0}, upload=197807]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 15:34:37.447415593 (190h8m31.878495724s) [upload=197807]: Done uploading segments... Aug 16 15:34:37.447444478 (190h8m31.878524609s) [upload=197807]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.249713024s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.230005926s FAIL (2025-08-16 15:34:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== M7V4VRQLWV6RRWXV6VIRN6YBALMPSQOUIUQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.841776238 (190h7m22.272856369s) [noise=true , address=104.36.82.50:4046 , piece_id=CWCROAFS5KRER423FT6RRFKN53V64DDYXUE6R53535I2JLQTELKQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.842091221 (190h7m22.273171352s) [noise=true , address=104.36.82.60:4366 , piece_id=HAFM65Y27IX6TWC6RRMPONRX7PR5OMRUIKMGC3BJKQ6IAED6PYWA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.842099617 (190h7m22.273179738s) [noise=true , address=77.90.31.13:4660 , piece_id=TUVSS6NDIX7VFTQQGKRHDVJDZPR75YQDRJUG6SEKKCCDMKG5JV7A, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.842362269 (190h7m22.2734424s) [noise=true , address=77.90.31.21:4887 , piece_id=FIYCC7KYDFUFH7R764SRMO6TFS54A2FRX7PMNROLZI6ZDZROG3QQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.842615785 (190h7m22.273695916s) [noise=true , address=104.36.82.35:4591 , piece_id=GCQ55CEDTND4G7ZU6AGCO7FZM6KWLYYUKPOHSWXXBM3IUMBPV6TA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.842970785 (190h7m22.274050916s) [noise=true , address=77.90.31.18:4837 , piece_id=F4XHRCVHYHFH3KC3CZEHO63KJBBN6XYHY6LDU23LVO2QEEGP6BZQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.843209601 (190h7m22.274289722s) [noise=true , address=104.36.82.5:4848 , piece_id=2QRIBYC65RJDOD2RZG5YZY5QMLH7HMBRT76NLB5RGJTC3Q52VZ6A, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.843424392 (190h7m22.274504513s) [noise=true , address=79.127.201.215:4359 , piece_id=G3ONFAVRHYB3VJLFWMUQOXTFSKV372UROLDX5DNW426AO36MGDDA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.843647469 (190h7m22.2747276s) [noise=true , address=45.91.116.14:4219 , piece_id=XFSXCD5NPGFKPRNRKP6T2YSD6334HWTV475TSWHVLBECM25VHNPQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.843871447 (190h7m22.274951568s) [noise=true , address=104.36.82.3:4780 , piece_id=SVP6AC436X27RLZIHIU3E2P75O7HOUDVAKTIW6VVMIWWZV3LAQJA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.844075137 (190h7m22.275155258s) [noise=true , address=104.36.82.23:4462 , piece_id=OTLBQ4YNZVHBFZYIJTUFFGVEHLBIXSW2SL5LOVNVDRPW32DMMGYA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.844265331 (190h7m22.275345462s) [noise=true , address=104.36.82.4:4830 , piece_id=5JH7VZCK5V2VQSB2SDX2DARLU3DP72NCDCGKAYAI4BKJKMMPRXCA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.844491133 (190h7m22.275571264s) [noise=true , address=79.127.205.235:4661 , piece_id=JDBGQMTFSTIDOXE4XHUCSCXOMR7OEDZWILHQCPIQNGQC7LJ3LZFA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.844673793 (190h7m22.275753914s) [noise=true , address=77.90.31.11:4567 , piece_id=JMP2TJL6JYYXOSRP7XFC7QVD3M23NONYBW3ISHR5Q57HOBSFAMGA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.846456865 (190h7m22.277536996s) [noise=true , address=104.36.82.44:4899 , piece_id=MGXNYLD7SMPFWWH5ERMLWOM4CY6ZFEW7I3YJYBGZSZ63XCKIKSFQ, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:27.851475968 (190h7m22.282556099s) [noise=true , address=77.90.31.15:4731 , piece_id=CXR7HEMW2DPHZ26MRV7SI3PUXKACVI4LOGC5LUE26RMX7BLVYCJA, seg_pos={0 0}, upload=197804]: Uploading piece... Aug 16 15:33:28.955760664 (190h7m23.386840796s) [upload=197804]: Next returned nil segment Aug 16 15:33:28.955777808 (190h7m23.386857929s) [upload=197804]: Waiting for error group managing segments... Aug 16 15:33:29.715037130 (190h7m24.146117261s) [noise=true , address=79.127.205.235:4661 , piece_id=JDBGQMTFSTIDOXE4XHUCSCXOMR7OEDZWILHQCPIQNGQC7LJ3LZFA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:30.054385852 (190h7m24.485465983s) [noise=true , address=79.127.201.214:4319 , piece_id=QSGAG4CDEPC4E4KO2M7V4VRQLWV6RRWXV6VIRN6YBALMPSQOUIUQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:30.899404879 (190h7m25.33048501s) [noise=true , address=104.36.82.50:4046 , piece_id=CWCROAFS5KRER423FT6RRFKN53V64DDYXUE6R53535I2JLQTELKQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:31.117001169 (190h7m25.54808129s) [noise=true , address=79.127.201.215:4359 , piece_id=G3ONFAVRHYB3VJLFWMUQOXTFSKV372UROLDX5DNW426AO36MGDDA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:34.692114679 (190h7m29.12319481s) [noise=true , address=77.90.31.18:4837 , piece_id=F4XHRCVHYHFH3KC3CZEHO63KJBBN6XYHY6LDU23LVO2QEEGP6BZQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:35.240458614 (190h7m29.671538745s) [noise=true , address=77.90.31.15:4731 , piece_id=CXR7HEMW2DPHZ26MRV7SI3PUXKACVI4LOGC5LUE26RMX7BLVYCJA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:39.757732601 (190h7m34.188812722s) [noise=true , address=77.90.31.12:4605 , piece_id=SFFBPWHKJQSLY5WVOBZQEQ5JFW6SX3QVK6U2ZQIWXWX6TLF5EECA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:41.899435372 (190h7m36.330515493s) [noise=true , address=77.90.31.11:4567 , piece_id=JMP2TJL6JYYXOSRP7XFC7QVD3M23NONYBW3ISHR5Q57HOBSFAMGA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:42.119089620 (190h7m36.550169751s) [noise=true , address=77.90.31.13:4660 , piece_id=TUVSS6NDIX7VFTQQGKRHDVJDZPR75YQDRJUG6SEKKCCDMKG5JV7A, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:33:42.352038156 (190h7m36.783118287s) [noise=true , address=45.91.116.14:4219 , piece_id=XFSXCD5NPGFKPRNRKP6T2YSD6334HWTV475TSWHVLBECM25VHNPQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: <nil> Aug 16 15:34:26.738372347 (190h8m21.169452468s) [noise=true , address=104.36.82.60:4366 , piece_id=HAFM65Y27IX6TWC6RRMPONRX7PR5OMRUIKMGC3BJKQ6IAED6PYWA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV, address:104.36.82.60:4366): protocol: rpc: dial tcp 104.36.82.60:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.738435428 (190h8m21.169515559s) [noise=true , address=77.90.31.21:4887 , piece_id=FIYCC7KYDFUFH7R764SRMO6TFS54A2FRX7PMNROLZI6ZDZROG3QQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: write tcp 188.166.225.170:39348->77.90.31.21:4887: use of closed network connection; write tcp 188.166.225.170:39348->77.90.31.21:4887: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39348->77.90.31.21:4887: use of closed network connection Aug 16 15:34:26.738469674 (190h8m21.169549795s) [noise=true , address=104.36.82.35:4591 , piece_id=GCQ55CEDTND4G7ZU6AGCO7FZM6KWLYYUKPOHSWXXBM3IUMBPV6TA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55, address:104.36.82.35:4591): protocol: write tcp 188.166.225.170:57340->104.36.82.35:4591: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742127142 (190h8m21.173207273s) [noise=true , address=104.36.82.5:4848 , piece_id=2QRIBYC65RJDOD2RZG5YZY5QMLH7HMBRT76NLB5RGJTC3Q52VZ6A, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:1xvod7N8eEwunLDYWAvVUP1Cp1CdEr3quJUMG5kWyBrwzzr9vp, address:104.36.82.5:4848): protocol: rpc: dial tcp 104.36.82.5:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742177428 (190h8m21.173257549s) [noise=true , address=104.36.82.23:4462 , piece_id=OTLBQ4YNZVHBFZYIJTUFFGVEHLBIXSW2SL5LOVNVDRPW32DMMGYA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742212355 (190h8m21.173292476s) [noise=true , address=104.36.82.4:4830 , piece_id=5JH7VZCK5V2VQSB2SDX2DARLU3DP72NCDCGKAYAI4BKJKMMPRXCA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:1p91PtS64Ue8JkPmbh2f3gig8cTQ3AMdDJPaP1249h4XmXUJrp, address:104.36.82.4:4830): protocol: rpc: dial tcp 104.36.82.4:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742254776 (190h8m21.173334907s) [noise=true , address=104.36.82.44:4899 , piece_id=MGXNYLD7SMPFWWH5ERMLWOM4CY6ZFEW7I3YJYBGZSZ63XCKIKSFQ, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: write tcp 188.166.225.170:33144->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742294672 (190h8m21.173374803s) [noise=true , address=104.36.82.3:4780 , piece_id=SVP6AC436X27RLZIHIU3E2P75O7HOUDVAKTIW6VVMIWWZV3LAQJA, seg_pos={0 0}, upload=197804]: Done uploading piece. err: ecclient: upload failed (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM, address:104.36.82.3:4780): protocol: write tcp 188.166.225.170:58490->104.36.82.3:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:26.742306946 (190h8m21.173387067s) [seg_pos={0 0}, upload=197804]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:34:26.966951362 (190h8m21.398031494s) [upload=197804]: Done uploading segments... Aug 16 15:34:26.966993082 (190h8m21.398073213s) [upload=197804]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230017007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X......X..XXX] duration: 30.226287502s FAIL (2025-08-16 15:34:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 16 15:33:43.520594440 (190h7m37.951674571s) [noise=true , address=77.90.31.21:4878 , piece_id=KNNMJRC7T7ZKZR2W6EMV4XCC2RKM2DMATGPGVNRIDJ4CPH3P5KNA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.520852634 (190h7m37.951932765s) [noise=true , address=104.36.82.6:4876 , piece_id=4PVGQLZ6WSIEYQAUI34BNWG67BWZAPZRRG57YBDEUMR5OITHQTPQ, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.521115617 (190h7m37.952195748s) [noise=true , address=104.36.82.14:4223 , piece_id=5WBOWC2S72ZKLNZUNBJD4MSFCS4YUBKEOGT75SKTL6QSZDHRC6MA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.521399059 (190h7m37.95247919s) [noise=true , address=104.36.82.12:4149 , piece_id=3YTUPHR3IQTDVQNK63OHR657B2I2L7V6VBZFECQRJO5DFDPJO6JQ, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.521652684 (190h7m37.952732816s) [noise=true , address=45.91.116.11:4119 , piece_id=L3B2AFWP3XYLE7ZJ5BDLIS2GX24OEIES3RFZ5562ME3A424TKP4A, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.521909927 (190h7m37.952990058s) [noise=true , address=104.36.82.7:4909 , piece_id=BF7FJINDDBVII2UVZWMELW56PK2MYUWWBO4PKYU54FZYRWOIFD7Q, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.522141630 (190h7m37.953221781s) [noise=true , address=45.91.116.14:4190 , piece_id=7VXWDKQJPVXQIBJ6CVGCIK2GKO4Y2P2DSJP5DHG45VDKOHP4WMPA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.522365809 (190h7m37.95344593s) [noise=true , address=104.36.82.15:4259 , piece_id=FPVKVVUPBLQGEKHTM5REKOZEMPHDK72YWGLC74EHEGHWHARYCVQQ, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.522554620 (190h7m37.953634751s) [noise=true , address=104.36.82.15:4257 , piece_id=K7SFQ6GIPXMKRDKOK5YHT7ZQFVVVUDIWESN4CM4CYWKYXRRZANRQ, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.522743061 (190h7m37.953823182s) [noise=true , address=104.36.82.10:4077 , piece_id=4WR4DFGLBLV6B3CYJI3TPAXGXBXY7PBLBDJNDKSCQOC6ZBKHCITA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.522935419 (190h7m37.95401554s) [noise=true , address=45.91.116.8:4048 , piece_id=LON6IJC3HPUV3RS3NR7AFKPP5CZUKBSJLKKUWE6QIAPNIEEECGHA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.523120183 (190h7m37.954200314s) [noise=true , address=104.36.82.5:4839 , piece_id=6Y4XPDXCJ5O2AVDL77T3M6VXGC4EB6Z6ID4NSKPQCV76CP5LR3CA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:43.523390120 (190h7m37.954470251s) [noise=true , address=104.36.82.12:4150 , piece_id=EM5NK32OPZY7SVO7PUKV4MSQ5ULTAIUC7P3YSPG436KYESYT33YA, seg_pos={0 0}, upload=197808]: Uploading piece... Aug 16 15:33:45.174915391 (190h7m39.605995522s) [upload=197808]: Next returned nil segment Aug 16 15:33:45.174929928 (190h7m39.606010049s) [upload=197808]: Waiting for error group managing segments... Aug 16 15:33:45.571447959 (190h7m40.00252809s) [noise=true , address=104.36.82.15:4259 , piece_id=FPVKVVUPBLQGEKHTM5REKOZEMPHDK72YWGLC74EHEGHWHARYCVQQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.613028288 (190h7m40.044108419s) [noise=true , address=104.36.82.10:4077 , piece_id=4WR4DFGLBLV6B3CYJI3TPAXGXBXY7PBLBDJNDKSCQOC6ZBKHCITA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.622170231 (190h7m40.053250362s) [noise=true , address=104.36.82.12:4149 , piece_id=3YTUPHR3IQTDVQNK63OHR657B2I2L7V6VBZFECQRJO5DFDPJO6JQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.646149072 (190h7m40.077229203s) [noise=true , address=104.36.82.15:4257 , piece_id=K7SFQ6GIPXMKRDKOK5YHT7ZQFVVVUDIWESN4CM4CYWKYXRRZANRQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.650342175 (190h7m40.081422306s) [noise=true , address=104.36.82.6:4876 , piece_id=4PVGQLZ6WSIEYQAUI34BNWG67BWZAPZRRG57YBDEUMR5OITHQTPQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.714148112 (190h7m40.145228243s) [noise=true , address=104.36.82.12:4150 , piece_id=EM5NK32OPZY7SVO7PUKV4MSQ5ULTAIUC7P3YSPG436KYESYT33YA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:33:45.750978393 (190h7m40.182058524s) [noise=true , address=104.36.82.7:4909 , piece_id=BF7FJINDDBVII2UVZWMELW56PK2MYUWWBO4PKYU54FZYRWOIFD7Q, seg_pos={0 0}, upload=197808]: Done uploading piece. err: <nil> Aug 16 15:34:12.635414229 (190h8m7.066494361s) [noise=true , address=45.91.116.9:4262 , piece_id=33D7G3KDINZRDIO3XQUJAUYK77N5BKLPJOSYIBVBAPEMY27ODYBQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 188.166.225.170:41922->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:41922->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41922->45.91.116.9:4262: use of closed network connection Aug 16 15:34:12.635464957 (190h8m7.066545088s) [noise=true , address=104.36.82.5:4839 , piece_id=6Y4XPDXCJ5O2AVDL77T3M6VXGC4EB6Z6ID4NSKPQCV76CP5LR3CA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:12.635495315 (190h8m7.066575446s) [noise=true , address=45.91.116.11:4119 , piece_id=L3B2AFWP3XYLE7ZJ5BDLIS2GX24OEIES3RFZ5562ME3A424TKP4A, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:47478->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:47478->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47478->45.91.116.11:4119: use of closed network connection Aug 16 15:34:12.635527967 (190h8m7.066608098s) [noise=true , address=45.91.116.8:4048 , piece_id=LON6IJC3HPUV3RS3NR7AFKPP5CZUKBSJLKKUWE6QIAPNIEEECGHA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:44588->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:44588->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->45.91.116.8:4048: use of closed network connection Aug 16 15:34:12.635552684 (190h8m7.066632815s) [noise=true , address=45.91.116.5:4299 , piece_id=XSTZQDHNEBO3DALYK3KTUEDIKYB3T2NGOOR55CQXJEH2KS3HDZPA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:43564->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:43564->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43564->45.91.116.5:4299: use of closed network connection Aug 16 15:34:12.635596048 (190h8m7.066676179s) [noise=true , address=77.90.31.21:4878 , piece_id=KNNMJRC7T7ZKZR2W6EMV4XCC2RKM2DMATGPGVNRIDJ4CPH3P5KNA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:57020->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:57020->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->77.90.31.21:4878: use of closed network connection Aug 16 15:34:12.635618751 (190h8m7.066698882s) [noise=true , address=45.91.116.14:4190 , piece_id=7VXWDKQJPVXQIBJ6CVGCIK2GKO4Y2P2DSJP5DHG45VDKOHP4WMPA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:48668->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:48668->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48668->45.91.116.14:4190: use of closed network connection Aug 16 15:34:12.635624071 (190h8m7.066704192s) [noise=true , address=104.36.82.14:4223 , piece_id=5WBOWC2S72ZKLNZUNBJD4MSFCS4YUBKEOGT75SKTL6QSZDHRC6MA, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:34:12.635633339 (190h8m7.06671347s) [noise=true , address=45.91.116.6:4973 , piece_id=XAGBB2RWFRHQ7AHDLFLD6TQXMEF23AOIAO3I4HQ4FCFUAFXFAOOQ, seg_pos={0 0}, upload=197808]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:50836->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50836->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50836->45.91.116.6:4973: use of closed network connection Aug 16 15:34:12.635648999 (190h8m7.06672913s) [seg_pos={0 0}, upload=197808]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Aug 16 15:34:12.860189586 (190h8m7.291269717s) [upload=197808]: Done uploading segments... Aug 16 15:34:12.860215115 (190h8m7.291295246s) [upload=197808]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.22630807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [X............X...X...X..................X.X..X.....X........] duration: 3.391401353s FAIL (2025-08-16 15:34:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 3.391433455s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X...........X......X...X.X......X..X......X..X...XXXXX] duration: 1m1.979827077s FAIL (2025-08-16 15:33:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:32:35.643857300 (190h6m30.074937431s) [upload=197790]: Starting upload Aug 16 15:32:35.643904280 (190h6m30.074984411s) [upload=197790]: Done starting upload Aug 16 15:32:35.643986598 (190h6m30.075066729s) [upload=197790]: Uploading segments... Aug 16 15:32:35.644077932 (190h6m30.075158063s) [upload=197790]: Got next segment. Inline: false Aug 16 15:32:35.644109042 (190h6m30.075189173s) [upload=197790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:32:35.644113420 (190h6m30.075193551s) [upload=197790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:32:36.399006518 (190h6m30.830086669s) [seg_pos={0 0}, upload=197790]: Begin upload segment... Aug 16 15:32:36.399054460 (190h6m30.830134591s) [seg_pos={0 0}, upload=197790]: Done begin upload segment. Aug 16 15:32:36.399076553 (190h6m30.830156674s) [noise=true , address=104.36.82.58:4296 , piece_id=GZ7J3BVY5IGGIN3ZNQXGBMOWYZSNFBTJRZP3TYYQN5FGAVJWP3SA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.399177556 (190h6m30.830257697s) [noise=true , address=77.90.31.18:4839 , piece_id=VOMGJK3CJJDFL7ICLXDGM6Z2U3LKYA4N5HESGPVWN4HR4B5EHT6Q, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.399500854 (190h6m30.830580985s) [noise=true , address=104.36.82.44:4873 , piece_id=6PUHBHOZWSZ6NP5CSF2AGPIIKBE3UIBRIVJTOY4FGQGMXHGFGVRA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.399800578 (190h6m30.830880709s) [noise=true , address=104.36.82.34:4554 , piece_id=5AAAP76WHP3JY663LCQSTELLQKMBGG6FAPVDBPDGPNFZUZJVNWMA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.399888416 (190h6m30.830968548s) [noise=true , address=104.36.82.5:4856 , piece_id=BZWL6A7SEMVHTQAJXEKMCIPQ6GF7VEOEWHCJT226YPH4TDBAXVMA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.400238916 (190h6m30.831319037s) [noise=true , address=77.90.31.12:4617 , piece_id=R3IJZ77DSUA25KUJ4CUIOGAOBO7F5IAZIIPVNNTOTD3SJAOCGEZQ, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.400338547 (190h6m30.831418678s) [noise=true , address=104.36.82.18:4372 , piece_id=EIU4QPWWCZAXXDCZXHHSBBECXCEOMHDWA6MIELUG7G4ZSJ6QTUXQ, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.400596030 (190h6m30.831676161s) [noise=true , address=79.127.201.214:4320 , piece_id=SB2ZJS55O4N2HNS7K6BLRSPNBAS6FZI3DD466RFK6JUF4RMRFPJA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.400754192 (190h6m30.831834313s) [noise=true , address=104.36.82.23:4464 , piece_id=WQGWKQA52MCVMKL34HZOJK3NDCK6ILISUV26E5QOGNB44G3Y3UBA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.400918126 (190h6m30.831998247s) [noise=true , address=45.91.116.12:4228 , piece_id=ZVQR5QPRAJJOKMWWAW5J4DFEM5NNRPHAU6NTFJBZT7LQJ6IZN2DQ, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.401238059 (190h6m30.8323182s) [noise=true , address=77.90.31.13:4648 , piece_id=LQCCWYEEKRWE6PJROXOWH5NXGLDE643YVDGA74YCWCPO57TUNKKA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.401261303 (190h6m30.832341434s) [noise=true , address=104.36.82.38:4696 , piece_id=C7HKVISPGZFBHW5SVHZZIDM54TFVB6HN5FR3KX7OE52IB2JM55EA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.401492906 (190h6m30.832573037s) [noise=true , address=77.90.31.11:4584 , piece_id=S5JA577PZST5P24532AJNHGEQK26XUY3US2QXZ3YO3OK7QQG4XHA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.401661959 (190h6m30.83274209s) [noise=true , address=104.36.82.36:4648 , piece_id=5IVNFZEK7YJZFJRGEWEHGCSSGHDAJLEK55HHKFAAYD447HSOZZBQ, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.401714851 (190h6m30.832794982s) [noise=true , address=104.36.82.60:4355 , piece_id=Z3QZAUKA3PRH4EQX2XHOKGLFC7ZZNEY4F24OUZXTZAPKWJH4PCZA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.402057927 (190h6m30.833138059s) [noise=true , address=104.36.82.39:4749 , piece_id=3OFEIHTFZRVHNUK7FWNCOA7A7DJ43MEUPSJZ52CXSNXD7PFIHO4Q, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.402093736 (190h6m30.833173867s) [noise=true , address=66.23.193.175:4560 , piece_id=LO3VDIT4TIUURMYUE5SAKSFUFGTUUX22H2YOJELROCTDQ4VEQC4A, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:36.405929706 (190h6m30.837009837s) [noise=true , address=77.90.31.15:4720 , piece_id=KV6FU6GY7VXNSLVUCBXT6W4I7AA4FKDXVGQHLZUAO3KVEF47PBFA, seg_pos={0 0}, upload=197790]: Uploading piece... Aug 16 15:32:37.477195874 (190h6m31.908275995s) [noise=true , address=79.127.201.214:4320 , piece_id=SB2ZJS55O4N2HNS7K6BLRSPNBAS6FZI3DD466RFK6JUF4RMRFPJA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:12ZLCXYP6sppXZcmj49C8jjgXbLGzgyowtpb1um9XZguzfVnM2r, address:79.127.201.214:4320): protocol: storage node overloaded, request limit: 2000 Aug 16 15:32:37.652778254 (190h6m32.083858385s) [upload=197790]: Next returned nil segment Aug 16 15:32:37.652796800 (190h6m32.083876931s) [upload=197790]: Waiting for error group managing segments... Aug 16 15:32:39.987923700 (190h6m34.419003831s) [noise=true , address=77.90.31.18:4839 , piece_id=VOMGJK3CJJDFL7ICLXDGM6Z2U3LKYA4N5HESGPVWN4HR4B5EHT6Q, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:32:40.018232072 (190h6m34.449312203s) [noise=true , address=77.90.31.11:4584 , piece_id=S5JA577PZST5P24532AJNHGEQK26XUY3US2QXZ3YO3OK7QQG4XHA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:32:40.621082701 (190h6m35.052162822s) [noise=true , address=77.90.31.13:4648 , piece_id=LQCCWYEEKRWE6PJROXOWH5NXGLDE643YVDGA74YCWCPO57TUNKKA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:32:44.153049233 (190h6m38.584129364s) [noise=true , address=66.23.193.175:4560 , piece_id=LO3VDIT4TIUURMYUE5SAKSFUFGTUUX22H2YOJELROCTDQ4VEQC4A, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:32:44.478277527 (190h6m38.909357658s) [noise=true , address=45.91.116.12:4228 , piece_id=ZVQR5QPRAJJOKMWWAW5J4DFEM5NNRPHAU6NTFJBZT7LQJ6IZN2DQ, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:32:51.827741860 (190h6m46.258821991s) [noise=true , address=104.36.82.5:4856 , piece_id=BZWL6A7SEMVHTQAJXEKMCIPQ6GF7VEOEWHCJT226YPH4TDBAXVMA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:33:11.295363995 (190h7m5.726444126s) [noise=true , address=77.90.31.15:4720 , piece_id=KV6FU6GY7VXNSLVUCBXT6W4I7AA4FKDXVGQHLZUAO3KVEF47PBFA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: <nil> Aug 16 15:33:34.987883565 (190h7m29.418963696s) [noise=true , address=77.90.31.12:4617 , piece_id=R3IJZ77DSUA25KUJ4CUIOGAOBO7F5IAZIIPVNNTOTD3SJAOCGEZQ, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: rpc: dial tcp 77.90.31.12:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.987929703 (190h7m29.419009834s) [noise=true , address=104.36.82.18:4372 , piece_id=EIU4QPWWCZAXXDCZXHHSBBECXCEOMHDWA6MIELUG7G4ZSJ6QTUXQ, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.987964920 (190h7m29.419045041s) [noise=true , address=104.36.82.23:4464 , piece_id=WQGWKQA52MCVMKL34HZOJK3NDCK6ILISUV26E5QOGNB44G3Y3UBA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:57484->104.36.82.23:4464: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988090762 (190h7m29.419170893s) [noise=true , address=104.36.82.39:4749 , piece_id=3OFEIHTFZRVHNUK7FWNCOA7A7DJ43MEUPSJZ52CXSNXD7PFIHO4Q, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:104.36.82.39:4749): protocol: rpc: dial tcp 104.36.82.39:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988159092 (190h7m29.419239213s) [noise=true , address=104.36.82.44:4873 , piece_id=6PUHBHOZWSZ6NP5CSF2AGPIIKBE3UIBRIVJTOY4FGQGMXHGFGVRA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: rpc: dial tcp 104.36.82.44:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988226561 (190h7m29.419306733s) [noise=true , address=104.36.82.60:4355 , piece_id=Z3QZAUKA3PRH4EQX2XHOKGLFC7ZZNEY4F24OUZXTZAPKWJH4PCZA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:104.36.82.60:4355): protocol: rpc: dial tcp 104.36.82.60:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988268451 (190h7m29.419348582s) [noise=true , address=104.36.82.34:4554 , piece_id=5AAAP76WHP3JY663LCQSTELLQKMBGG6FAPVDBPDGPNFZUZJVNWMA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:12rSDkPXm47w73hry9W6BWwAiwNpB3cJuwedNt1JuSEEtd3Sup7, address:104.36.82.34:4554): protocol: rpc: dial tcp 104.36.82.34:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988375386 (190h7m29.419455517s) [noise=true , address=104.36.82.38:4696 , piece_id=C7HKVISPGZFBHW5SVHZZIDM54TFVB6HN5FR3KX7OE52IB2JM55EA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988414972 (190h7m29.419495153s) [noise=true , address=104.36.82.58:4296 , piece_id=GZ7J3BVY5IGGIN3ZNQXGBMOWYZSNFBTJRZP3TYYQN5FGAVJWP3SA, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:12nctdBBFSDaAV2c5PUbZW64K3v9CH3a49NSfTh6dta4DVxtqXA, address:104.36.82.58:4296): protocol: write tcp 188.166.225.170:38230->104.36.82.58:4296: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988426153 (190h7m29.419506284s) [noise=true , address=104.36.82.36:4648 , piece_id=5IVNFZEK7YJZFJRGEWEHGCSSGHDAJLEK55HHKFAAYD447HSOZZBQ, seg_pos={0 0}, upload=197790]: Done uploading piece. err: ecclient: upload failed (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq, address:104.36.82.36:4648): protocol: rpc: dial tcp 104.36.82.36:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:34.988442875 (190h7m29.419523006s) [seg_pos={0 0}, upload=197790]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:33:36.966215731 (190h7m31.397295862s) [upload=197790]: Done uploading segments... Aug 16 15:33:36.966240869 (190h7m31.397321s) [upload=197790]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m1.979847796s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.230335857s FAIL (2025-08-16 15:33:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== PPF52VRPEZHRAVBVGVVMQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.353672496 (190h6m21.784752627s) [noise=true , address=104.36.82.36:4649 , piece_id=WGPWY2QAIWBUMHYJLU7BF5TEBV7XRTZSAN73EFIXJRIH4U5PJJUA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.353758501 (190h6m21.784838632s) [noise=true , address=104.36.82.9:4025 , piece_id=CULRGYNUTSMNRVEN4VDYL22U6P3YLS2PUD6XT7XGIFPS6W4ATYDA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354085727 (190h6m21.785165858s) [noise=true , address=45.91.116.14:4219 , piece_id=EQ6BVRGKVQSAHBAG6JJGHDZBJXC6ZK5FXZC45HTQFUEWBUPI4SIQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354201138 (190h6m21.785281269s) [noise=true , address=77.90.31.19:4850 , piece_id=Y2OCJUKYFNFRI2UJVGRTAOTPF5YWXVGTYTCXPHDES4XKOASC272A, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354381593 (190h6m21.785461724s) [noise=true , address=66.23.193.175:4564 , piece_id=LHTGXSOIHEDZZPF7F27HQDMBIB4BU4RRCKUCSJWBJQDHPC4G4JKQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354562789 (190h6m21.78564291s) [noise=true , address=77.90.31.16:4754 , piece_id=DX2V4E5TABPYJRAR5A6EZBKRSAJF7WALCA44BJMPZOAOYE3LMW5Q, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354688731 (190h6m21.785768862s) [noise=true , address=45.91.116.12:4233 , piece_id=ZFIV5676ZZDEM75P45AQ4TXU36RBKDWZPVO2TRIPZMJC2JJRQ64Q, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354924793 (190h6m21.786004924s) [noise=true , address=104.36.82.60:4353 , piece_id=QRC3GSFUI4A5ONYRSQ3QXVEICGJL24UHAR5UNV37UWOJLLCTHWAQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.354968275 (190h6m21.786048406s) [noise=true , address=104.36.82.38:4712 , piece_id=ZQLYZZSGTCPFWP5NCCGOBZJM25R2F75RQHJ4CHMCBJJCYIVKNDZA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.355278308 (190h6m21.786358439s) [noise=true , address=104.36.82.5:4861 , piece_id=R5OIIFPTRLW3OQZQJYC5ABIZMCMUHEUF24KNMN2CNHV6G5ITVTEA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.356169234 (190h6m21.787249365s) [noise=true , address=104.36.82.23:4448 , piece_id=NNRM7QUQ2JQ5YD2GQQUZUYRBZFN2NWY6PUXOXWUL6R3UTRFUGFOQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.356536776 (190h6m21.787616907s) [noise=true , address=104.36.82.44:4893 , piece_id=NT5XMMOR3G7IFAM73YVW5QQHRKMI7MYB6YDOORY3VEWRO7TRG5UQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.356884893 (190h6m21.787965024s) [noise=true , address=66.23.193.175:4560 , piece_id=HOQWY73B2RARFTY5H4PRD5GVYDDQS66GZXQ2N5QBDQYOJUZKVWBA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.357202601 (190h6m21.788282732s) [noise=true , address=104.36.82.34:4572 , piece_id=5EO2DDU5H7G773F24FVHJKVCFEIIEXJFOKFBYKZP6OVP5ZCC6VIA, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.357494689 (190h6m21.78857482s) [noise=true , address=104.36.82.59:4335 , piece_id=MYYCNXFUCIW2BQKZYUSG4RGE464SOGLAPRPKUT7YNFILIC6ILYHQ, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:27.358392006 (190h6m21.789472128s) [noise=true , address=45.91.116.4:4950 , piece_id=RJG24UTMZXQERIKCP3TGPMQ4C66K6ORRL6VLOOJ6VEUEJ7AEDC2A, seg_pos={0 0}, upload=197788]: Uploading piece... Aug 16 15:32:28.493057625 (190h6m22.924137756s) [upload=197788]: Next returned nil segment Aug 16 15:32:28.493074317 (190h6m22.924154448s) [upload=197788]: Waiting for error group managing segments... Aug 16 15:32:29.019163741 (190h6m23.450243872s) [noise=true , address=66.23.193.175:4564 , piece_id=LHTGXSOIHEDZZPF7F27HQDMBIB4BU4RRCKUCSJWBJQDHPC4G4JKQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:33.315983899 (190h6m27.74706403s) [noise=true , address=77.90.31.19:4850 , piece_id=Y2OCJUKYFNFRI2UJVGRTAOTPF5YWXVGTYTCXPHDES4XKOASC272A, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:34.044125681 (190h6m28.475205812s) [noise=true , address=45.91.116.12:4233 , piece_id=ZFIV5676ZZDEM75P45AQ4TXU36RBKDWZPVO2TRIPZMJC2JJRQ64Q, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:35.709700205 (190h6m30.140780336s) [noise=true , address=45.91.116.6:4985 , piece_id=6ZVW4OXYVOQ25GSV6IXM5IEYY6WOMO7PPF52VRPEZHRAVBVGVVMQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:36.311600405 (190h6m30.742680536s) [noise=true , address=66.23.193.175:4560 , piece_id=HOQWY73B2RARFTY5H4PRD5GVYDDQS66GZXQ2N5QBDQYOJUZKVWBA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:36.747477876 (190h6m31.178558007s) [noise=true , address=77.90.31.16:4754 , piece_id=DX2V4E5TABPYJRAR5A6EZBKRSAJF7WALCA44BJMPZOAOYE3LMW5Q, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:37.008569210 (190h6m31.439649331s) [noise=true , address=45.91.116.4:4950 , piece_id=RJG24UTMZXQERIKCP3TGPMQ4C66K6ORRL6VLOOJ6VEUEJ7AEDC2A, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:40.719362801 (190h6m35.150442932s) [noise=true , address=79.127.201.215:4355 , piece_id=TJCZT3XGVAX7TGKWB3U7EB2VMGDRW4HITSB3ABUDQAYBII7HONGQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:32:43.482011585 (190h6m37.913091716s) [noise=true , address=45.91.116.14:4219 , piece_id=EQ6BVRGKVQSAHBAG6JJGHDZBJXC6ZK5FXZC45HTQFUEWBUPI4SIQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: <nil> Aug 16 15:33:26.250499478 (190h7m20.681579599s) [noise=true , address=104.36.82.23:4448 , piece_id=NNRM7QUQ2JQ5YD2GQQUZUYRBZFN2NWY6PUXOXWUL6R3UTRFUGFOQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i, address:104.36.82.23:4448): protocol: rpc: dial tcp 104.36.82.23:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250586494 (190h7m20.681666625s) [noise=true , address=104.36.82.59:4335 , piece_id=MYYCNXFUCIW2BQKZYUSG4RGE464SOGLAPRPKUT7YNFILIC6ILYHQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:12gFW6PBz447rjBeSsnrdskgBcCHrqvB1SDCcLBzHU8G972fUm2, address:104.36.82.59:4335): protocol: rpc: dial tcp 104.36.82.59:4335: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250623474 (190h7m20.681703605s) [noise=true , address=104.36.82.9:4025 , piece_id=CULRGYNUTSMNRVEN4VDYL22U6P3YLS2PUD6XT7XGIFPS6W4ATYDA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 188.166.225.170:36614->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250770566 (190h7m20.681850697s) [noise=true , address=104.36.82.36:4649 , piece_id=WGPWY2QAIWBUMHYJLU7BF5TEBV7XRTZSAN73EFIXJRIH4U5PJJUA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: write tcp 188.166.225.170:48286->104.36.82.36:4649: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250802117 (190h7m20.681882248s) [noise=true , address=104.36.82.5:4861 , piece_id=R5OIIFPTRLW3OQZQJYC5ABIZMCMUHEUF24KNMN2CNHV6G5ITVTEA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: rpc: dial tcp 104.36.82.5:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250837955 (190h7m20.681918086s) [noise=true , address=104.36.82.38:4712 , piece_id=ZQLYZZSGTCPFWP5NCCGOBZJM25R2F75RQHJ4CHMCBJJCYIVKNDZA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250845450 (190h7m20.681925581s) [noise=true , address=104.36.82.44:4893 , piece_id=NT5XMMOR3G7IFAM73YVW5QQHRKMI7MYB6YDOORY3VEWRO7TRG5UQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: rpc: dial tcp 104.36.82.44:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250882030 (190h7m20.681962161s) [noise=true , address=104.36.82.34:4572 , piece_id=5EO2DDU5H7G773F24FVHJKVCFEIIEXJFOKFBYKZP6OVP5ZCC6VIA, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:124jj7LpTJR3kta3gK57SzNwATM9NuKpNCwkiC2zzedeH7YBAf3, address:104.36.82.34:4572): protocol: rpc: dial tcp 104.36.82.34:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250930072 (190h7m20.682010213s) [noise=true , address=104.36.82.60:4353 , piece_id=QRC3GSFUI4A5ONYRSQ3QXVEICGJL24UHAR5UNV37UWOJLLCTHWAQ, seg_pos={0 0}, upload=197788]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:26.250963085 (190h7m20.682043216s) [seg_pos={0 0}, upload=197788]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:33:26.477963439 (190h7m20.90904357s) [upload=197788]: Done uploading segments... Aug 16 15:33:26.477995650 (190h7m20.909075831s) [upload=197788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230351638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.X............X...X...X..................X.X..X.....X.......] duration: 5.104828317s FAIL (2025-08-16 15:33:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 5.104852313s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X......X..XX] duration: 30.231468222s FAIL (2025-08-16 15:33:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:32:39.247053213 (190h6m33.678133344s) [upload=197792]: Starting upload Aug 16 15:32:39.247164586 (190h6m33.678244717s) [upload=197792]: Done starting upload Aug 16 15:32:39.247256803 (190h6m33.678336934s) [upload=197792]: Uploading segments... Aug 16 15:32:39.247862031 (190h6m33.678942162s) [upload=197792]: Got next segment. Inline: false Aug 16 15:32:39.247931303 (190h6m33.679011454s) [upload=197792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:32:39.247938678 (190h6m33.679018809s) [upload=197792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:32:39.487349037 (190h6m33.918429168s) [seg_pos={0 0}, upload=197792]: Begin upload segment... Aug 16 15:32:39.487388583 (190h6m33.918468714s) [seg_pos={0 0}, upload=197792]: Done begin upload segment. Aug 16 15:32:39.487410855 (190h6m33.918490986s) [noise=true , address=77.90.31.12:4592 , piece_id=CK7WMNXHY3KNQPWWJG53QCCK2ERJEHVHT4K7XRUSBHVWOQKJ5RZA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.487466462 (190h6m33.918546593s) [noise=true , address=104.36.82.14:4222 , piece_id=YZPOZPXJL5ASAHPANH727ELAVIXIYW6EOLGHGCNWU7ID36G3FEIA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.487780222 (190h6m33.918860343s) [noise=true , address=45.91.116.4:4938 , piece_id=6KUNZHYTSBKOGE6UZ4ICA4O4HXRVH2UH7VDZNTT255CSET3XAXFQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.487983191 (190h6m33.919063322s) [noise=true , address=104.36.82.6:4876 , piece_id=USYGBR5OH522TGZESAVIVCU6QHYMERXNMXCRSXZ6MUABWR7MFBXA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488052663 (190h6m33.919132794s) [noise=true , address=77.90.31.14:4663 , piece_id=JMQNTEVOTFYHPBLD7LNN4KJXVBMPUCHSFCUUX5WYXTEK36K4BA7Q, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488354861 (190h6m33.919434992s) [noise=true , address=77.90.31.19:4843 , piece_id=G2LHPDJ6EW2OUWVZH7N5JGAGREQ6DB26HXXN2RMS2AYF24UMFJ3Q, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488374810 (190h6m33.919454941s) [noise=true , address=45.91.116.6:4973 , piece_id=S5NIP474XEH7TBVYLTOGNMNJH357QPDAFCKIF54TMMHSCFOUV7JA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488671387 (190h6m33.919751518s) [noise=true , address=104.36.82.15:4257 , piece_id=I2BGOXXU45T472MXUFIZZ2HEGMRW5NHOIF4QDIKLIA4NVGJ32TOQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488711303 (190h6m33.919791424s) [noise=true , address=104.36.82.11:4115 , piece_id=ZS7VNL2L776PJ7TFT7UWUJMTQUU4DYFVZOQI72TPHK4HAPSTFEXA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.488941514 (190h6m33.920021645s) [noise=true , address=45.91.116.11:4117 , piece_id=ZZI5YIHQM5YQNB7WTXNQDBC5C6O5MZCHAKTGFD2XC7J56TWPXSTQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489043799 (190h6m33.92012393s) [noise=true , address=104.36.82.10:4077 , piece_id=LLTNHCVZW22ZV6CS3R4CFY2XVIYWLHDXWHAGD6WLI4ATDCUY2HPA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489161916 (190h6m33.920242037s) [noise=true , address=104.36.82.13:4186 , piece_id=GCTKEC52USAPF24JNVGH7UVWK3FZ6CQTYKKPSWHSUGEKK4SMVGRA, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489289650 (190h6m33.920369771s) [noise=true , address=77.90.31.18:4805 , piece_id=ZEBLP3ZFWJBM3FDR7VENEPFBTGKODDRBYNZXEASCGRYT2YGAPR2Q, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489368531 (190h6m33.920448662s) [noise=true , address=77.90.31.11:4554 , piece_id=LL7O2GJJAL7GEEY5ELUA5XBWIOAT7EBHIOAHUSWZUPIXNPYF7BGQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489537324 (190h6m33.920617455s) [noise=true , address=45.91.116.5:4298 , piece_id=UHMVKMTZ5WNIENPA7QCT5UXCDTBYHLW2DW7KDTEN5VHH3PTN3ZMQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:39.489598762 (190h6m33.920678893s) [noise=true , address=104.36.82.10:4078 , piece_id=5JZDC6BRZLLY2X7COHX54BXXFIWXNAT2NJDDQVQFV7I7SSN2AHFQ, seg_pos={0 0}, upload=197792]: Uploading piece... Aug 16 15:32:41.249660547 (190h6m35.680740678s) [upload=197792]: Next returned nil segment Aug 16 15:32:41.249677309 (190h6m35.68075744s) [upload=197792]: Waiting for error group managing segments... Aug 16 15:32:41.541353164 (190h6m35.972433295s) [noise=true , address=104.36.82.15:4257 , piece_id=I2BGOXXU45T472MXUFIZZ2HEGMRW5NHOIF4QDIKLIA4NVGJ32TOQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:41.598770968 (190h6m36.029851099s) [noise=true , address=104.36.82.10:4078 , piece_id=5JZDC6BRZLLY2X7COHX54BXXFIWXNAT2NJDDQVQFV7I7SSN2AHFQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:41.608906111 (190h6m36.039986242s) [noise=true , address=104.36.82.6:4876 , piece_id=USYGBR5OH522TGZESAVIVCU6QHYMERXNMXCRSXZ6MUABWR7MFBXA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:41.681530050 (190h6m36.112610181s) [noise=true , address=104.36.82.10:4077 , piece_id=LLTNHCVZW22ZV6CS3R4CFY2XVIYWLHDXWHAGD6WLI4ATDCUY2HPA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.260505767 (190h6m36.691585888s) [noise=true , address=45.91.116.11:4117 , piece_id=ZZI5YIHQM5YQNB7WTXNQDBC5C6O5MZCHAKTGFD2XC7J56TWPXSTQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.303899336 (190h6m36.734979497s) [noise=true , address=45.91.116.4:4938 , piece_id=6KUNZHYTSBKOGE6UZ4ICA4O4HXRVH2UH7VDZNTT255CSET3XAXFQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.475975718 (190h6m36.907055849s) [noise=true , address=77.90.31.18:4805 , piece_id=ZEBLP3ZFWJBM3FDR7VENEPFBTGKODDRBYNZXEASCGRYT2YGAPR2Q, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.728534486 (190h6m37.159614617s) [noise=true , address=77.90.31.11:4554 , piece_id=LL7O2GJJAL7GEEY5ELUA5XBWIOAT7EBHIOAHUSWZUPIXNPYF7BGQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.874329952 (190h6m37.305410083s) [noise=true , address=45.91.116.6:4973 , piece_id=S5NIP474XEH7TBVYLTOGNMNJH357QPDAFCKIF54TMMHSCFOUV7JA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:42.893328624 (190h6m37.324408755s) [noise=true , address=45.91.116.5:4298 , piece_id=UHMVKMTZ5WNIENPA7QCT5UXCDTBYHLW2DW7KDTEN5VHH3PTN3ZMQ, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:43.179383484 (190h6m37.610463605s) [noise=true , address=77.90.31.14:4663 , piece_id=JMQNTEVOTFYHPBLD7LNN4KJXVBMPUCHSFCUUX5WYXTEK36K4BA7Q, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:32:52.758203631 (190h6m47.189283792s) [noise=true , address=77.90.31.19:4843 , piece_id=G2LHPDJ6EW2OUWVZH7N5JGAGREQ6DB26HXXN2RMS2AYF24UMFJ3Q, seg_pos={0 0}, upload=197792]: Done uploading piece. err: <nil> Aug 16 15:33:08.597258359 (190h7m3.02833849s) [noise=true , address=104.36.82.11:4115 , piece_id=ZS7VNL2L776PJ7TFT7UWUJMTQUU4DYFVZOQI72TPHK4HAPSTFEXA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:40470->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:08.597332561 (190h7m3.028412692s) [noise=true , address=104.36.82.13:4186 , piece_id=GCTKEC52USAPF24JNVGH7UVWK3FZ6CQTYKKPSWHSUGEKK4SMVGRA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:57990->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:08.597384070 (190h7m3.028464201s) [noise=true , address=77.90.31.12:4592 , piece_id=CK7WMNXHY3KNQPWWJG53QCCK2ERJEHVHT4K7XRUSBHVWOQKJ5RZA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:36860->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:08.597398737 (190h7m3.028478869s) [noise=true , address=104.36.82.14:4222 , piece_id=YZPOZPXJL5ASAHPANH727ELAVIXIYW6EOLGHGCNWU7ID36G3FEIA, seg_pos={0 0}, upload=197792]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44028->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:08.597425960 (190h7m3.028506091s) [seg_pos={0 0}, upload=197792]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:33:08.827570139 (190h7m3.25865026s) [upload=197792]: Done uploading segments... Aug 16 15:33:08.827594636 (190h7m3.258674767s) [upload=197792]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.231486847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [..X............X...X...X..................X.X..X.....X......] duration: 5.41667112s FAIL (2025-08-16 15:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 5.416691999s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.228607019s FAIL (2025-08-16 15:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_ObjectBegin Aug 16 15:31:26.425831198 (190h5m20.856911329s) [upload=197774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:31:26.872589859 (190h5m21.30366999s) [seg_pos={0 0}, upload=197774]: Begin upload segment... Aug 16 15:31:26.872649153 (190h5m21.303729294s) [seg_pos={0 0}, upload=197774]: Done begin upload segment. Aug 16 15:31:26.872682166 (190h5m21.303762297s) [noise=true , address=104.36.82.23:4449 , piece_id=R7DOZQUXHJRZ46B24B6KSHKMKESXYOM2FK7TFDIHJJJUKJIEO3TA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.872707694 (190h5m21.303787826s) [noise=true , address=104.36.82.60:4355 , piece_id=3WCXNEZYXLI5IGFSPZ2MSPF3QO5SQPYUQBYT4WHFLBL2YLI6HZYA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873061932 (190h5m21.304142063s) [noise=true , address=45.91.116.9:4270 , piece_id=ZXXBDXRTR5C6P5SEX6J53FMPLX6OALUWSQL245JPVQCMCKFLUVYQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873168877 (190h5m21.304249018s) [noise=true , address=66.23.193.175:4563 , piece_id=4JULC7ZZYJESQCDVMTACDP5XVJAP7ZJ3KPHWGCYDRHSKJFTKSJCQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873316260 (190h5m21.304396391s) [noise=true , address=104.36.82.34:4576 , piece_id=XIF4UQTELJ4JSN5HXWCOBFJICHYNTHW7LAMSFD6TH3H3XPY7V3NQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873526932 (190h5m21.304607063s) [noise=true , address=104.36.82.2:4752 , piece_id=PTSZUXYKI2MCP57H6U6TB6GC26I5HVQMNGEXYXDI5KS5LZTPMIWA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873578381 (190h5m21.304658512s) [noise=true , address=79.127.201.214:4319 , piece_id=NF5FTQ3SY5OYUBCCA4DZBAKCCAYNYPRTT54ECPC7OWYAMTUJBDFA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.873968968 (190h5m21.305049099s) [noise=true , address=104.36.82.18:4389 , piece_id=WWNIUFNFAD7JIP5ZSPGDLMMKHO4SRLBZPQSNE3KOPG2A7CMTZDNA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.874969733 (190h5m21.306049864s) [noise=true , address=104.36.82.44:4891 , piece_id=UEY4SNZ5L75UHJZD2RBBY5G6HVJLQWHNYH4GIO4DVDAR2IZMRBEQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.876161533 (190h5m21.307241664s) [noise=true , address=79.127.201.215:4357 , piece_id=H2THQYKEXLQAOSXXTTIB6742MVAQZ63LCL33EYQTA6R6MBPS4WXA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.876363109 (190h5m21.30744324s) [noise=true , address=77.90.31.10:4527 , piece_id=TY73LNCSRBPWSRFMJTUPSKXWOGPSE7WMNZZ2M3A3O3EX3JUT5XSQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.876745651 (190h5m21.307825792s) [noise=true , address=77.90.31.19:4875 , piece_id=W2HRUAN6CHATQMFRLQI7ZR6UAS2SONZPJE24EN2EAW2QGO3ONLNQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.877039072 (190h5m21.308119203s) [noise=true , address=77.90.31.13:4660 , piece_id=W6W42RPKAKLC2FGU35AS6JI4HICJQFGZY2LTW72XMFXQOFNOU5VA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.877283831 (190h5m21.308363962s) [noise=true , address=77.90.31.14:4683 , piece_id=NCXHFKZE2BKFVC7JNHZT2RL2HABUDLX3F2K46GD7EZ522SFUAGSQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.877531845 (190h5m21.308611976s) [noise=true , address=104.36.82.5:4866 , piece_id=LEDGSRPHJZ2Y37FZTZQMHRJXKDSAJOS4XH3LENSHKSGELKP3VLOQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.878338790 (190h5m21.309418921s) [noise=true , address=79.127.201.215:4349 , piece_id=6DAE6RS6UAWJQW7W6FJOX2EEUVMOJE46CYJHFYEDE4CT2YFMDM7A, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.879493118 (190h5m21.310573249s) [noise=true , address=77.90.31.11:4567 , piece_id=FYIYFNABS2XGZZQHCQDZ4D6ZQGYRBR6UCYZMNKJV5FLOZNIEY3NA, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:26.879752244 (190h5m21.310832375s) [noise=true , address=104.36.82.58:4266 , piece_id=Z4KEI7G5LAYPIPBKVSFQZJKLJ4IR7WK4WIDQ3VR5K6PFUQXN34EQ, seg_pos={0 0}, upload=197774]: Uploading piece... Aug 16 15:31:28.056992334 (190h5m22.488072465s) [upload=197774]: Next returned nil segment Aug 16 15:31:28.057009096 (190h5m22.488089227s) [upload=197774]: Waiting for error group managing segments... Aug 16 15:31:30.142310277 (190h5m24.573390408s) [noise=true , address=79.127.201.215:4349 , piece_id=6DAE6RS6UAWJQW7W6FJOX2EEUVMOJE46CYJHFYEDE4CT2YFMDM7A, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:30.421670858 (190h5m24.852750989s) [noise=true , address=79.127.201.215:4357 , piece_id=H2THQYKEXLQAOSXXTTIB6742MVAQZ63LCL33EYQTA6R6MBPS4WXA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:31.339105735 (190h5m25.770185866s) [noise=true , address=79.127.201.214:4319 , piece_id=NF5FTQ3SY5OYUBCCA4DZBAKCCAYNYPRTT54ECPC7OWYAMTUJBDFA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:31.587875903 (190h5m26.018956034s) [noise=true , address=66.23.193.175:4563 , piece_id=4JULC7ZZYJESQCDVMTACDP5XVJAP7ZJ3KPHWGCYDRHSKJFTKSJCQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:36.717913097 (190h5m31.148993228s) [noise=true , address=104.36.82.58:4266 , piece_id=Z4KEI7G5LAYPIPBKVSFQZJKLJ4IR7WK4WIDQ3VR5K6PFUQXN34EQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:49.378173597 (190h5m43.809253728s) [noise=true , address=45.91.116.9:4270 , piece_id=ZXXBDXRTR5C6P5SEX6J53FMPLX6OALUWSQL245JPVQCMCKFLUVYQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:55.883303943 (190h5m50.314384074s) [noise=true , address=77.90.31.19:4875 , piece_id=W2HRUAN6CHATQMFRLQI7ZR6UAS2SONZPJE24EN2EAW2QGO3ONLNQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:31:58.767863982 (190h5m53.198944113s) [noise=true , address=77.90.31.14:4683 , piece_id=NCXHFKZE2BKFVC7JNHZT2RL2HABUDLX3F2K46GD7EZ522SFUAGSQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:32:00.982721326 (190h5m55.413801458s) [noise=true , address=104.36.82.2:4752 , piece_id=PTSZUXYKI2MCP57H6U6TB6GC26I5HVQMNGEXYXDI5KS5LZTPMIWA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:32:03.024569064 (190h5m57.455649195s) [noise=true , address=77.90.31.13:4660 , piece_id=W6W42RPKAKLC2FGU35AS6JI4HICJQFGZY2LTW72XMFXQOFNOU5VA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:32:05.414578114 (190h5m59.845658245s) [noise=true , address=77.90.31.11:4567 , piece_id=FYIYFNABS2XGZZQHCQDZ4D6ZQGYRBR6UCYZMNKJV5FLOZNIEY3NA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:32:24.923591000 (190h6m19.354671131s) [noise=true , address=77.90.31.10:4527 , piece_id=TY73LNCSRBPWSRFMJTUPSKXWOGPSE7WMNZZ2M3A3O3EX3JUT5XSQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: <nil> Aug 16 15:32:25.785921860 (190h6m20.217001991s) [noise=true , address=104.36.82.18:4389 , piece_id=WWNIUFNFAD7JIP5ZSPGDLMMKHO4SRLBZPQSNE3KOPG2A7CMTZDNA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:1mMqCXPNzFx4cnu8Nb3gNKcJFJ4tGJjnkwY2EVPjUBnwNVcM9R, address:104.36.82.18:4389): protocol: write tcp 188.166.225.170:56526->104.36.82.18:4389: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:25.785964982 (190h6m20.217045103s) [noise=true , address=104.36.82.23:4449 , piece_id=R7DOZQUXHJRZ46B24B6KSHKMKESXYOM2FK7TFDIHJJJUKJIEO3TA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:25.786062970 (190h6m20.217143101s) [noise=true , address=104.36.82.44:4891 , piece_id=UEY4SNZ5L75UHJZD2RBBY5G6HVJLQWHNYH4GIO4DVDAR2IZMRBEQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:34014->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:25.786174584 (190h6m20.217254715s) [noise=true , address=104.36.82.60:4355 , piece_id=3WCXNEZYXLI5IGFSPZ2MSPF3QO5SQPYUQBYT4WHFLBL2YLI6HZYA, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:104.36.82.60:4355): protocol: rpc: dial tcp 104.36.82.60:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:25.786203589 (190h6m20.21728372s) [noise=true , address=104.36.82.34:4576 , piece_id=XIF4UQTELJ4JSN5HXWCOBFJICHYNTHW7LAMSFD6TH3H3XPY7V3NQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus, address:104.36.82.34:4576): protocol: write tcp 188.166.225.170:36670->104.36.82.34:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:25.786275145 (190h6m20.217355276s) [noise=true , address=104.36.82.5:4866 , piece_id=LEDGSRPHJZ2Y37FZTZQMHRJXKDSAJOS4XH3LENSHKSGELKP3VLOQ, seg_pos={0 0}, upload=197774]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:32:25.786289193 (190h6m20.217369314s) [seg_pos={0 0}, upload=197774]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:32:26.012335411 (190h6m20.443415582s) [upload=197774]: Done uploading segments... Aug 16 15:32:26.012361220 (190h6m20.443441351s) [upload=197774]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.228618119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [...X............X...X...X..................X.X..X.....X.....] duration: 7.955265036s FAIL (2025-08-16 15:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 7.955297539s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X...........X......X...X.X......X..X......X..X...XXXX] duration: 1m0.909728802s FAIL (2025-08-16 15:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:31:20.817905891 (190h5m15.248986022s) [upload=197771]: Starting upload Aug 16 15:31:20.817970644 (190h5m15.249050775s) [upload=197771]: Done starting upload Aug 16 15:31:20.818119970 (190h5m15.249200091s) [upload=197771]: Uploading segments... Aug 16 15:31:20.818277913 (190h5m15.249358044s) [upload=197771]: Got next segment. Inline: false Aug 16 15:31:20.818311497 (190h5m15.249391618s) [upload=197771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:31:20.818316105 (190h5m15.249396237s) [upload=197771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:31:21.863660826 (190h5m16.294740957s) [seg_pos={0 0}, upload=197771]: Begin upload segment... Aug 16 15:31:21.863707665 (190h5m16.294787796s) [seg_pos={0 0}, upload=197771]: Done begin upload segment. Aug 16 15:31:21.863729377 (190h5m16.294809508s) [noise=true , address=104.36.82.44:4902 , piece_id=45JUWNWDLQRZK2RWU3JQMZHHLCJ7J46NTRV22NB3GZ5KVRXTHUGA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.863766478 (190h5m16.294846619s) [noise=true , address=104.36.82.39:4731 , piece_id=ZBUSZLWY5ND3BFNU3WCR5H7YIKFPCE2WNOY7IHKMJEJMAQKNG7FQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.864111839 (190h5m16.29519197s) [noise=true , address=77.90.31.10:4529 , piece_id=UTKX4SHO2DJ4HFLZWHXACTFFLOIFEUOXUSZZCY6AUTOG727AQUUQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.864455887 (190h5m16.295536018s) [noise=true , address=77.90.31.19:4853 , piece_id=7UEYZFLORRGMS7XDO5S2FVB547CNKHMUP6Z3KP5V67KEQGO3BTXA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.864536802 (190h5m16.295616933s) [noise=true , address=66.23.193.175:4559 , piece_id=ZRJ7OHKMV4XGM7DTR22FPJ2GFUNDXQV2ZA5E35HCZWVUZ2NSNROQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.864753176 (190h5m16.295833307s) [noise=true , address=79.127.201.215:4357 , piece_id=RY6DBBJKY5Y3GCE7NU4Y6RKY6KFFS5S7WSI4OUXN4EOJZ2QCFVVQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.865033432 (190h5m16.296113563s) [noise=true , address=104.36.82.5:4851 , piece_id=6FBVRQYIVDS34TTPQLC4W4TWFAJVJ37F3SR3KF6PUQEEHMG6OB3Q, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.865070493 (190h5m16.296150624s) [noise=true , address=104.36.82.60:4347 , piece_id=BF2CVRDWS34UAMZTHTVD6ZZB5CF34LPTN3FYN33WUXASMRTU2DQA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.865332875 (190h5m16.296413006s) [noise=true , address=79.127.219.51:4198 , piece_id=EG2ZUXIXRB4AG4PWYKTDNB75ICL4DTXWZFL5SSH626V7PNTQDWXA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.865653889 (190h5m16.29673402s) [noise=true , address=104.36.82.23:4454 , piece_id=Z4F2OSFHNQRPPCHKC7WEGFTFXATNV6TZP5GQ7HQCTDT3R6MDSNSQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.865967639 (190h5m16.29704776s) [noise=true , address=77.90.31.15:4711 , piece_id=ISNYYQCNV3DTRIRFOOB6BWZVE532TDD7GH7LPTI546G2VDNFWNMQ, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.866239970 (190h5m16.297320101s) [noise=true , address=77.90.31.11:4573 , piece_id=JCTLVS27PGMBEXMAUH3YJWMK7S6GKAPWPKSEFH7XGFRPGL6FGSTA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.866515467 (190h5m16.297595599s) [noise=true , address=64.49.13.155:4922 , piece_id=WKTVWEKJGV5RX66VW3PZBSYUBNRLSMICTEQTVNQO6IJUL3UNUXVA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.866850670 (190h5m16.297930801s) [noise=true , address=104.36.82.38:4715 , piece_id=LNGZENTCWRHGIM744KTIB3RCMYAMVDECNH3EC2HEML6UZOA7CXHA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.867080008 (190h5m16.298160139s) [noise=true , address=104.36.82.35:4613 , piece_id=KZBVUSRBBDEYBIRCKIV7XFOPAVN4XUPBC3CA4COGVTYZGZ6VQPEA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.867297815 (190h5m16.298377946s) [noise=true , address=77.90.31.21:4888 , piece_id=DQ6D7WFJ5SZQTTJB3SGM5NDPRBDNNLH4X3VQOG6BX3B5RWD5SG3Q, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.867524869 (190h5m16.298605s) [noise=true , address=104.36.82.59:4335 , piece_id=23LN4BGNMJT4KGUOIEF65TONLLPSQQ4XKYRWJ5YBHPYAUA73KEKA, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:21.867739500 (190h5m16.298819631s) [noise=true , address=104.36.82.9:4025 , piece_id=CVVKLSS374SUHVQWPH4Q34Y2HBPQQCWUQHNPV3EF2TM3JPHWHO5A, seg_pos={0 0}, upload=197771]: Uploading piece... Aug 16 15:31:23.032526343 (190h5m17.463606474s) [upload=197771]: Next returned nil segment Aug 16 15:31:23.032543875 (190h5m17.463624007s) [upload=197771]: Waiting for error group managing segments... Aug 16 15:31:23.494484837 (190h5m17.925564968s) [noise=true , address=64.49.13.155:4922 , piece_id=WKTVWEKJGV5RX66VW3PZBSYUBNRLSMICTEQTVNQO6IJUL3UNUXVA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:26.573898080 (190h5m21.004978211s) [noise=true , address=79.127.201.215:4357 , piece_id=RY6DBBJKY5Y3GCE7NU4Y6RKY6KFFS5S7WSI4OUXN4EOJZ2QCFVVQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:27.202533636 (190h5m21.633613767s) [noise=true , address=79.127.219.51:4198 , piece_id=EG2ZUXIXRB4AG4PWYKTDNB75ICL4DTXWZFL5SSH626V7PNTQDWXA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:27.453481001 (190h5m21.884561132s) [noise=true , address=104.36.82.9:4025 , piece_id=CVVKLSS374SUHVQWPH4Q34Y2HBPQQCWUQHNPV3EF2TM3JPHWHO5A, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:29.412813784 (190h5m23.843893915s) [noise=true , address=66.23.193.175:4559 , piece_id=ZRJ7OHKMV4XGM7DTR22FPJ2GFUNDXQV2ZA5E35HCZWVUZ2NSNROQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:33.943153771 (190h5m28.374233942s) [noise=true , address=77.90.31.21:4888 , piece_id=DQ6D7WFJ5SZQTTJB3SGM5NDPRBDNNLH4X3VQOG6BX3B5RWD5SG3Q, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:38.186077824 (190h5m32.617157955s) [noise=true , address=77.90.31.10:4529 , piece_id=UTKX4SHO2DJ4HFLZWHXACTFFLOIFEUOXUSZZCY6AUTOG727AQUUQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:34136->77.90.31.10:4529: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:39.833617139 (190h5m34.26469727s) [noise=true , address=77.90.31.15:4711 , piece_id=ISNYYQCNV3DTRIRFOOB6BWZVE532TDD7GH7LPTI546G2VDNFWNMQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:31:51.773276315 (190h5m46.204356446s) [noise=true , address=77.90.31.19:4853 , piece_id=7UEYZFLORRGMS7XDO5S2FVB547CNKHMUP6Z3KP5V67KEQGO3BTXA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:32:17.203502492 (190h6m11.634582623s) [noise=true , address=77.90.31.11:4573 , piece_id=JCTLVS27PGMBEXMAUH3YJWMK7S6GKAPWPKSEFH7XGFRPGL6FGSTA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: <nil> Aug 16 15:32:20.167572448 (190h6m14.598652579s) [noise=true , address=104.36.82.5:4851 , piece_id=6FBVRQYIVDS34TTPQLC4W4TWFAJVJ37F3SR3KF6PUQEEHMG6OB3Q, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:104.36.82.5:4851): protocol: rpc: dial tcp 104.36.82.5:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167597085 (190h6m14.598677216s) [noise=true , address=104.36.82.35:4613 , piece_id=KZBVUSRBBDEYBIRCKIV7XFOPAVN4XUPBC3CA4COGVTYZGZ6VQPEA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1xjm7XPPwM9nXBP3SS36HeLNxBVJdEpcQyD9BAp79xbGCze4vL, address:104.36.82.35:4613): protocol: rpc: dial tcp 104.36.82.35:4613: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167611983 (190h6m14.598692114s) [noise=true , address=104.36.82.23:4454 , piece_id=Z4F2OSFHNQRPPCHKC7WEGFTFXATNV6TZP5GQ7HQCTDT3R6MDSNSQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2, address:104.36.82.23:4454): protocol: rpc: dial tcp 104.36.82.23:4454: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167636239 (190h6m14.59871637s) [noise=true , address=104.36.82.38:4715 , piece_id=LNGZENTCWRHGIM744KTIB3RCMYAMVDECNH3EC2HEML6UZOA7CXHA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: write tcp 188.166.225.170:55760->104.36.82.38:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167640447 (190h6m14.598720578s) [noise=true , address=104.36.82.39:4731 , piece_id=ZBUSZLWY5ND3BFNU3WCR5H7YIKFPCE2WNOY7IHKMJEJMAQKNG7FQ, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:104.36.82.39:4731): protocol: rpc: dial tcp 104.36.82.39:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167665746 (190h6m14.598745867s) [noise=true , address=104.36.82.59:4335 , piece_id=23LN4BGNMJT4KGUOIEF65TONLLPSQQ4XKYRWJ5YBHPYAUA73KEKA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:12gFW6PBz447rjBeSsnrdskgBcCHrqvB1SDCcLBzHU8G972fUm2, address:104.36.82.59:4335): protocol: write tcp 188.166.225.170:57974->104.36.82.59:4335: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167669633 (190h6m14.598749764s) [noise=true , address=104.36.82.44:4902 , piece_id=45JUWNWDLQRZK2RWU3JQMZHHLCJ7J46NTRV22NB3GZ5KVRXTHUGA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167674353 (190h6m14.598754484s) [noise=true , address=104.36.82.60:4347 , piece_id=BF2CVRDWS34UAMZTHTVD6ZZB5CF34LPTN3FYN33WUXASMRTU2DQA, seg_pos={0 0}, upload=197771]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: rpc: dial tcp 104.36.82.60:4347: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:20.167686375 (190h6m14.598766506s) [seg_pos={0 0}, upload=197771]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:32:21.075831838 (190h6m15.506911979s) [upload=197771]: Done uploading segments... Aug 16 15:32:21.075859731 (190h6m15.506939852s) [upload=197771]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.909750874s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X......X..X] duration: 30.235900904s FAIL (2025-08-16 15:32:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 78]: Uploading piece... Aug 16 15:31:37.644281276 (190h5m32.075361407s) [noise=true , address=104.36.82.12:4151 , piece_id=36VCB4WXJVMMU6TJGBMMPNXHCGPIZQJZ4QPEC6MIBYARY5P37FQQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.644461221 (190h5m32.075541342s) [noise=true , address=45.91.116.5:4299 , piece_id=TKF6G3JHSS5IOLX25E3VHFDJCDO4WHVHS4CGQECO4XYHTAIRR4EA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.644534861 (190h5m32.075614992s) [noise=true , address=45.91.116.8:4047 , piece_id=CNLMJVLSD3H7HS2SPCWFRGZZBIYETNOHLN5X4LLSQATUMQPXT22A, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.644728091 (190h5m32.075808222s) [noise=true , address=104.36.82.14:4223 , piece_id=DSUS52VPKVVVSTEK2OFVXUVY7FB4QZHUBIGJEPN6STFVX3GZRYGQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.644763629 (190h5m32.07584376s) [noise=true , address=77.90.31.10:4519 , piece_id=NURT2MEF2A5U3XI2LZXTKRVDTWP66YW2XJLX6XQOVFGSF6CZ4TGQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.644916732 (190h5m32.075996863s) [noise=true , address=104.36.82.7:4911 , piece_id=EI5UHIQCCARLDALPYSTL3QNILKB4TEFWZO5PUNKOKCFMZXCCOLEQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645085074 (190h5m32.076165215s) [noise=true , address=104.36.82.2:4729 , piece_id=P2DWMSBGF3WJX7CECAV2YIPNSGZVDRHW3ZXZD6XB3HPGGLI4YTFQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645124911 (190h5m32.076205042s) [noise=true , address=104.36.82.15:4259 , piece_id=7VCLWYDF2IOQ6QMIURMVXOFJSXUUY2N4PLBQ7QLNPCNUDOPSOXOA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645336385 (190h5m32.076416517s) [noise=true , address=104.36.82.6:4876 , piece_id=ZLTQVSQHDKW7ACWFAYSPXKZ2Z27TRFK6G7TA53WVB2KSO7K5B3CA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645431607 (190h5m32.076511738s) [noise=true , address=104.36.82.16:4295 , piece_id=GJ6QT222N2AB52WPBWACMKEETYRD6ZGBNTE5GJJBRBMKFBKSVMSQ, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645531629 (190h5m32.07661176s) [noise=true , address=104.36.82.10:4077 , piece_id=D3P72CHAORCZAUREQOFLPG34266CZF62Z5XOSISM4FDJ2QMENXQA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645691695 (190h5m32.076771826s) [noise=true , address=77.90.31.21:4878 , piece_id=VMMW7ZZGCJS4BQPXL2TPJ2SU7I3OHH3TJRWLA673FPZPH66F5RVA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.645732834 (190h5m32.076812965s) [noise=true , address=45.91.116.11:4119 , piece_id=2B3R7KPWVKFSKQMNHPKSZN4YKN64W2NQRHUPYRD6BZHOEI5X45HA, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:37.646321410 (190h5m32.077401541s) [noise=true , address=77.90.31.16:4733 , piece_id=QWXHGW7UN2NBAOYPWR5LGCVIBCCFBYGY5FWSLVU4Z4DKIYHN567Q, seg_pos={0 0}, upload=197778]: Uploading piece... Aug 16 15:31:39.471872865 (190h5m33.902952996s) [upload=197778]: Next returned nil segment Aug 16 15:31:39.471894024 (190h5m33.902974155s) [upload=197778]: Waiting for error group managing segments... Aug 16 15:31:39.753932239 (190h5m34.18501237s) [noise=true , address=104.36.82.15:4259 , piece_id=7VCLWYDF2IOQ6QMIURMVXOFJSXUUY2N4PLBQ7QLNPCNUDOPSOXOA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:39.798204379 (190h5m34.22928451s) [noise=true , address=104.36.82.6:4876 , piece_id=ZLTQVSQHDKW7ACWFAYSPXKZ2Z27TRFK6G7TA53WVB2KSO7K5B3CA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:39.832572430 (190h5m34.263652561s) [noise=true , address=104.36.82.10:4077 , piece_id=D3P72CHAORCZAUREQOFLPG34266CZF62Z5XOSISM4FDJ2QMENXQA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:39.918141779 (190h5m34.34922191s) [noise=true , address=104.36.82.12:4151 , piece_id=36VCB4WXJVMMU6TJGBMMPNXHCGPIZQJZ4QPEC6MIBYARY5P37FQQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:42.071851864 (190h5m36.502931995s) [noise=true , address=104.36.82.7:4911 , piece_id=EI5UHIQCCARLDALPYSTL3QNILKB4TEFWZO5PUNKOKCFMZXCCOLEQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:48.853067865 (190h5m43.284147996s) [noise=true , address=45.91.116.5:4299 , piece_id=TKF6G3JHSS5IOLX25E3VHFDJCDO4WHVHS4CGQECO4XYHTAIRR4EA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:31:55.662787245 (190h5m50.093867366s) [noise=true , address=104.36.82.2:4729 , piece_id=P2DWMSBGF3WJX7CECAV2YIPNSGZVDRHW3ZXZD6XB3HPGGLI4YTFQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: <nil> Aug 16 15:32:06.738521047 (190h6m1.169601178s) [noise=true , address=45.91.116.8:4047 , piece_id=CNLMJVLSD3H7HS2SPCWFRGZZBIYETNOHLN5X4LLSQATUMQPXT22A, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:59404->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:59404->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59404->45.91.116.8:4047: use of closed network connection Aug 16 15:32:06.738840789 (190h6m1.16992092s) [noise=true , address=45.91.116.11:4119 , piece_id=2B3R7KPWVKFSKQMNHPKSZN4YKN64W2NQRHUPYRD6BZHOEI5X45HA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:32:06.739001427 (190h6m1.170081548s) [noise=true , address=77.90.31.10:4519 , piece_id=NURT2MEF2A5U3XI2LZXTKRVDTWP66YW2XJLX6XQOVFGSF6CZ4TGQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:34874->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:34874->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34874->77.90.31.10:4519: use of closed network connection Aug 16 15:32:06.739910045 (190h6m1.170990176s) [noise=true , address=104.36.82.16:4295 , piece_id=GJ6QT222N2AB52WPBWACMKEETYRD6ZGBNTE5GJJBRBMKFBKSVMSQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:06.739979437 (190h6m1.171059568s) [noise=true , address=77.90.31.21:4878 , piece_id=VMMW7ZZGCJS4BQPXL2TPJ2SU7I3OHH3TJRWLA673FPZPH66F5RVA, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:39074->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:39074->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39074->77.90.31.21:4878: use of closed network connection Aug 16 15:32:06.740080822 (190h6m1.171160953s) [noise=true , address=104.36.82.14:4223 , piece_id=DSUS52VPKVVVSTEK2OFVXUVY7FB4QZHUBIGJEPN6STFVX3GZRYGQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:32:06.740136208 (190h6m1.171216339s) [noise=true , address=77.90.31.16:4733 , piece_id=QWXHGW7UN2NBAOYPWR5LGCVIBCCFBYGY5FWSLVU4Z4DKIYHN567Q, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:51348->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:51348->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51348->77.90.31.16:4733: use of closed network connection Aug 16 15:32:06.740217603 (190h6m1.171297734s) [noise=true , address=77.90.31.11:4553 , piece_id=CECZURF25MGQQ3CJAHVQ6UMQR7US3XT7UCWBDIANDCQYPD3LOVZQ, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48712->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48712->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48712->77.90.31.11:4553: use of closed network connection Aug 16 15:32:06.740240247 (190h6m1.171320378s) [noise=true , address=45.91.116.10:4083 , piece_id=5OFQ2337PKSIZUT5HGBZL53WSB3IRE3YISTUCUAQSFFJLOTUIJ3A, seg_pos={0 0}, upload=197778]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:40246->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:40246->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40246->45.91.116.10:4083: use of closed network connection Aug 16 15:32:06.740282768 (190h6m1.171362899s) [seg_pos={0 0}, upload=197778]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Aug 16 15:32:06.973235271 (190h6m1.404315403s) [upload=197778]: Done uploading segments... Aug 16 15:32:06.973259428 (190h6m1.404339559s) [upload=197778]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.235921254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [....X............X...X...X..................X.X..X.....X....] duration: 5.730449306s FAIL (2025-08-16 15:31:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 5.730493831s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.226637962s FAIL (2025-08-16 15:31:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 0.901172698s) [noise=true , address=104.36.82.39:4729 , piece_id=PMEGUUFKXUWBFUDGG6GSWL5GS5KXAUHJ4M4O6IRTRYRAE5UTRI2A, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.470184864 (190h4m20.901264985s) [noise=true , address=77.90.31.11:4573 , piece_id=I4LRWXOMRK4B7GXSDANAKOALLS73Q7TWD3BFLKQUINUUJ6QTOSFA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.470422047 (190h4m20.901502178s) [noise=true , address=104.36.82.23:4449 , piece_id=2THJWVT7AXY4AZ376FMMRDUDWXCY3TNQLA3TYIYCQGYIXVQXLUCA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.470758562 (190h4m20.901838693s) [noise=true , address=104.36.82.37:4688 , piece_id=FBHCE26OSNH2QKIDBU5IGWOHJU5GFIFWTRVDOHQYPK3CWQ2OQXZA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.472115227 (190h4m20.903195358s) [noise=true , address=66.23.193.175:4559 , piece_id=KEY766D5O3TKIEY4YCJLLHRQ4OCYJQMGALAFFJSYGPZNGIHCIBPA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.472481848 (190h4m20.903561979s) [noise=true , address=77.90.31.14:4672 , piece_id=ITF2HLQBGONRBFZDRBQGP3U5OK3CW4QO575NPKR2EZGQ6TB5MDVQ, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.472818994 (190h4m20.903899125s) [noise=true , address=104.36.82.17:4329 , piece_id=4H5OV57I7APGTN3PDU6KZEKATY4IYSRMVSSUTJE5RVLPYV36NOWA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.473125250 (190h4m20.904205381s) [noise=true , address=45.91.116.7:4017 , piece_id=FQNQC3BDH2A7ZF6ERNIZT65XB55YVKJDLWB25OBYKT34D2BHSAYQ, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.473356251 (190h4m20.904436382s) [noise=true , address=104.36.82.5:4866 , piece_id=2TXTEVGDQMHGONAYZOLXT4HDS223TGFY32XU7SKGYB5HE7IBCWHA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.473609827 (190h4m20.904689958s) [noise=true , address=77.90.31.13:4656 , piece_id=WMZBVIPIFT4JJUOE6FUB7PZFSZZHJUITHFIVWUIRQBQYCJF6GU7A, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.473913729 (190h4m20.90499386s) [noise=true , address=104.36.82.60:4340 , piece_id=DW7VOV4NXYVO26FZ6KH6K5TB7IEITDKPECRSB6I4VNZRSTNWYBOQ, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.474199044 (190h4m20.905279185s) [noise=true , address=45.91.116.5:4307 , piece_id=HAACDXZEKYTOTUX4UKIK4J7AZLNTKP7HSDTAZE467SEXASZSB4UA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.474541139 (190h4m20.90562127s) [noise=true , address=77.90.31.10:4542 , piece_id=JFXML24F4NNOBQLA22DYJLTPWMUGQSDSL6XSWS6B2YUIGCVYKLZA, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:26.474758866 (190h4m20.905838997s) [noise=true , address=104.36.82.44:4903 , piece_id=6LSMLOYPU3VWZFWVKARPDO5UL5MKUVXVNA2NAQ6SESHBHLTBK3UQ, seg_pos={0 0}, upload=197759]: Uploading piece... Aug 16 15:30:27.621359911 (190h4m22.052440032s) [upload=197759]: Next returned nil segment Aug 16 15:30:27.621376973 (190h4m22.052457094s) [upload=197759]: Waiting for error group managing segments... Aug 16 15:30:28.958488502 (190h4m23.389568633s) [noise=true , address=77.90.31.13:4656 , piece_id=WMZBVIPIFT4JJUOE6FUB7PZFSZZHJUITHFIVWUIRQBQYCJF6GU7A, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:29.181298209 (190h4m23.61237834s) [noise=true , address=45.91.116.5:4307 , piece_id=HAACDXZEKYTOTUX4UKIK4J7AZLNTKP7HSDTAZE467SEXASZSB4UA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:29.762983911 (190h4m24.194064032s) [noise=true , address=77.90.31.14:4672 , piece_id=ITF2HLQBGONRBFZDRBQGP3U5OK3CW4QO575NPKR2EZGQ6TB5MDVQ, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:31.030378944 (190h4m25.461459075s) [noise=true , address=45.91.116.7:4017 , piece_id=FQNQC3BDH2A7ZF6ERNIZT65XB55YVKJDLWB25OBYKT34D2BHSAYQ, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:34.821838852 (190h4m29.252918973s) [noise=true , address=66.23.193.175:4559 , piece_id=KEY766D5O3TKIEY4YCJLLHRQ4OCYJQMGALAFFJSYGPZNGIHCIBPA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:35.250305123 (190h4m29.681385254s) [noise=true , address=77.90.31.11:4573 , piece_id=I4LRWXOMRK4B7GXSDANAKOALLS73Q7TWD3BFLKQUINUUJ6QTOSFA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:30:36.396145413 (190h4m30.827225564s) [noise=true , address=104.36.82.5:4866 , piece_id=2TXTEVGDQMHGONAYZOLXT4HDS223TGFY32XU7SKGYB5HE7IBCWHA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:54878->104.36.82.5:4866: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:38.037874930 (190h4m32.468955061s) [noise=true , address=79.127.201.215:4353 , piece_id=C4MLB5ZLSSSI6LSAXJABCDHB4PKZ6VH32SHS2BYLZMKPDQKBIMVQ, seg_pos={0 0}, upload=197759]: Done uploading piece. err: <nil> Aug 16 15:31:25.338057763 (190h5m19.769137894s) [noise=true , address=104.36.82.60:4340 , piece_id=DW7VOV4NXYVO26FZ6KH6K5TB7IEITDKPECRSB6I4VNZRSTNWYBOQ, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338108871 (190h5m19.769189002s) [noise=true , address=77.90.31.10:4542 , piece_id=JFXML24F4NNOBQLA22DYJLTPWMUGQSDSL6XSWS6B2YUIGCVYKLZA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338146263 (190h5m19.769226394s) [noise=true , address=104.36.82.44:4903 , piece_id=6LSMLOYPU3VWZFWVKARPDO5UL5MKUVXVNA2NAQ6SESHBHLTBK3UQ, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: rpc: dial tcp 104.36.82.44:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338185628 (190h5m19.769265759s) [noise=true , address=104.36.82.23:4449 , piece_id=2THJWVT7AXY4AZ376FMMRDUDWXCY3TNQLA3TYIYCQGYIXVQXLUCA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338221436 (190h5m19.769301557s) [noise=true , address=104.36.82.36:4622 , piece_id=WMCQQVLN7ZE4XS56WUAGZGLRU4CJLOD7EMZRVXONVTQOFEEHSBYA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:104.36.82.36:4622): protocol: rpc: dial tcp 104.36.82.36:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338262696 (190h5m19.769342817s) [noise=true , address=104.36.82.35:4588 , piece_id=K7XTWSWEPYRE5REJAAVI43OQSCRWPPW6XH65YZCEBOHLDLOABFQA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5, address:104.36.82.35:4588): protocol: rpc: dial tcp 104.36.82.35:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338289888 (190h5m19.769370019s) [noise=true , address=104.36.82.39:4729 , piece_id=PMEGUUFKXUWBFUDGG6GSWL5GS5KXAUHJ4M4O6IRTRYRAE5UTRI2A, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:104.36.82.39:4729): protocol: write tcp 188.166.225.170:35774->104.36.82.39:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338298474 (190h5m19.769378595s) [noise=true , address=104.36.82.37:4688 , piece_id=FBHCE26OSNH2QKIDBU5IGWOHJU5GFIFWTRVDOHQYPK3CWQ2OQXZA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:12XHxPHd22b7s5tj2pYXzUnC6AXMnPnNSB4jw3RYfKtwu6jwaVN, address:104.36.82.37:4688): protocol: write tcp 188.166.225.170:57040->104.36.82.37:4688: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338321188 (190h5m19.769401309s) [noise=true , address=104.36.82.59:4328 , piece_id=6AKMAHMEU2MEHGOSYXYJFWM5EMARPJAEOBF6OWEOC6KBZVU55DAA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: rpc: dial tcp 104.36.82.59:4328: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338351395 (190h5m19.769431516s) [noise=true , address=104.36.82.17:4329 , piece_id=4H5OV57I7APGTN3PDU6KZEKATY4IYSRMVSSUTJE5RVLPYV36NOWA, seg_pos={0 0}, upload=197759]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:104.36.82.17:4329): protocol: write tcp 188.166.225.170:46934->104.36.82.17:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:25.338376734 (190h5m19.769456865s) [seg_pos={0 0}, upload=197759]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:31:25.562142058 (190h5m19.993222189s) [upload=197759]: Done uploading segments... Aug 16 15:31:25.562171915 (190h5m19.993252046s) [upload=197759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.226651698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.......X...........X......X...X.X......X..X......X..X...XXX] duration: 1m0.440733634s FAIL (2025-08-16 15:31:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:30:20.153313153 (190h4m14.584393284s) [upload=197756]: Starting upload Aug 16 15:30:20.153352968 (190h4m14.584433099s) [upload=197756]: Done starting upload Aug 16 15:30:20.153476004 (190h4m14.584556135s) [upload=197756]: Uploading segments... Aug 16 15:30:20.153610031 (190h4m14.584690252s) [upload=197756]: Got next segment. Inline: false Aug 16 15:30:20.153662662 (190h4m14.584742853s) [upload=197756]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:30:20.153673101 (190h4m14.584753302s) [upload=197756]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:30:20.664335657 (190h4m15.095415788s) [seg_pos={0 0}, upload=197756]: Begin upload segment... Aug 16 15:30:20.664383018 (190h4m15.095463149s) [seg_pos={0 0}, upload=197756]: Done begin upload segment. Aug 16 15:30:20.664400411 (190h4m15.095480532s) [noise=true , address=45.91.116.4:4950 , piece_id=YXWBUDDFIXNEVZKC3AKKPDJQ2NHBRF3FHXTV2LP5MTHS3MBLQNKQ, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.664408356 (190h4m15.095488487s) [noise=true , address=66.23.193.175:4554 , piece_id=UZXDNR2DYYZXQ65QYFOPWWZ5OERDL4ZHSCPR7JESXHEHPRVLKYKQ, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.664766772 (190h4m15.095846893s) [noise=true , address=104.36.82.60:4339 , piece_id=ZNNTY27WP7LYNREGOQWKV2QSNDZITGLD3AZLRCMUQCQPOWHFYZTA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665047208 (190h4m15.096127329s) [noise=true , address=104.36.82.38:4722 , piece_id=EHKZOC6PXHOMXCA4HG27GPFEXJBVXPPYVTRZU647MBG3UCNLYECA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665176075 (190h4m15.096256206s) [noise=true , address=104.36.82.59:4315 , piece_id=O6QPUXRKSQ62TR4CTNSFR6P53B6TGWQZRA6SPWXKYI3O4ANZTZ2A, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665292357 (190h4m15.096372488s) [noise=true , address=77.90.31.10:4539 , piece_id=ZBLKK5Q2WAKP62GX5OESK2GGKYXPZA6FKWB5O44E36MZC67BK22Q, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665467022 (190h4m15.096547143s) [noise=true , address=104.36.82.36:4622 , piece_id=MVNGNTCN7GIJTEYHHEMQZZFYBGOESACFGQNPFJ3LN7P26QRECBYQ, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665600998 (190h4m15.096681129s) [noise=true , address=45.91.116.12:4239 , piece_id=SMIPBBOWI6VTB3ILYSRN6Q6WH46DOMH24HDKHSWNBVFMXRRUHYQQ, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.665740766 (190h4m15.096820897s) [noise=true , address=79.127.205.235:4689 , piece_id=H2GPXFN3FNYJQPWT5NPP2DBVJDOFJ5LWW4OQV3U4M2KEPPYDEZCA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.666088801 (190h4m15.097168922s) [noise=true , address=77.90.31.15:4702 , piece_id=EZ7GJ4T3OTPXVQXYNK37P64WKYNIUYL6MYZ6AJTUQBMKI26ABGNA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.666416849 (190h4m15.09749697s) [noise=true , address=104.36.82.44:4872 , piece_id=UNJU6SUJBWGBGHQZPJYEYYENMZDJYMCOQWAFZBWMEOR6IKMKNJ7A, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.666557779 (190h4m15.0976379s) [noise=true , address=104.36.82.34:4563 , piece_id=B56ANLGWZHTURNMPTNCFDRBAC73EJR7LEDUNWUXHMI7VZTDGHXBA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.666773952 (190h4m15.097854083s) [noise=true , address=104.36.82.23:4450 , piece_id=KGN27KGUY5ZQ4H5BGDJAG3HWEZJD6O7XEEQWQ6IW4OHACJEJH5ZQ, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.666996528 (190h4m15.098076659s) [noise=true , address=77.90.31.12:4597 , piece_id=5AE5OVHHONAQHOJQRGOL2463RNZ4URKFL2JRLHJJDONVMUBJPQLA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.667194207 (190h4m15.098274328s) [noise=true , address=79.127.201.215:4359 , piece_id=R3E6RSP73BR4GSAXD2AE2AL3ZOAVLDOZBNXRA7JV6AHCHCX7OSLA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.667451579 (190h4m15.0985317s) [noise=true , address=79.127.220.101:4071 , piece_id=XIPL5IYOCMFU5CH43RHFUZ7W7VFMBGBOBU5DWMAFPID5Z3KAIM2Q, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.667588541 (190h4m15.098668672s) [noise=true , address=77.90.31.14:4688 , piece_id=5RCAY32XV4MXIU4FYNF5GIUCDVE3NTOD3SP37KKQN4LXH5ZZKDEA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:20.667807170 (190h4m15.098887301s) [noise=true , address=77.90.31.19:4873 , piece_id=WA7AG2KP6YKZGMEFOKEZXGWV36HXGW2VONYWKCQIDGQKXJMXKLAA, seg_pos={0 0}, upload=197756]: Uploading piece... Aug 16 15:30:21.806232271 (190h4m16.237312392s) [upload=197756]: Next returned nil segment Aug 16 15:30:21.806249364 (190h4m16.237329495s) [upload=197756]: Waiting for error group managing segments... Aug 16 15:30:22.601736415 (190h4m17.032816546s) [noise=true , address=79.127.205.235:4689 , piece_id=H2GPXFN3FNYJQPWT5NPP2DBVJDOFJ5LWW4OQV3U4M2KEPPYDEZCA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:22.671829401 (190h4m17.102909532s) [noise=true , address=79.127.201.215:4359 , piece_id=R3E6RSP73BR4GSAXD2AE2AL3ZOAVLDOZBNXRA7JV6AHCHCX7OSLA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:22.873635959 (190h4m17.30471609s) [noise=true , address=79.127.220.101:4071 , piece_id=XIPL5IYOCMFU5CH43RHFUZ7W7VFMBGBOBU5DWMAFPID5Z3KAIM2Q, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:23.097000656 (190h4m17.528080787s) [noise=true , address=66.23.193.175:4554 , piece_id=UZXDNR2DYYZXQ65QYFOPWWZ5OERDL4ZHSCPR7JESXHEHPRVLKYKQ, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:1CraGuuckkbKe5PDLUvwF5G2ATUbgdR5g192YjkZgmP3JpqgXi, address:66.23.193.175:4554): protocol: storage node overloaded, request limit: 2000 Aug 16 15:30:24.442504605 (190h4m18.873584736s) [noise=true , address=77.90.31.12:4597 , piece_id=5AE5OVHHONAQHOJQRGOL2463RNZ4URKFL2JRLHJJDONVMUBJPQLA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:24.947178031 (190h4m19.378258172s) [noise=true , address=45.91.116.12:4239 , piece_id=SMIPBBOWI6VTB3ILYSRN6Q6WH46DOMH24HDKHSWNBVFMXRRUHYQQ, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:25.045025743 (190h4m19.476105874s) [noise=true , address=77.90.31.19:4873 , piece_id=WA7AG2KP6YKZGMEFOKEZXGWV36HXGW2VONYWKCQIDGQKXJMXKLAA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:27.912909615 (190h4m22.343989746s) [noise=true , address=45.91.116.4:4950 , piece_id=YXWBUDDFIXNEVZKC3AKKPDJQ2NHBRF3FHXTV2LP5MTHS3MBLQNKQ, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:28.661673916 (190h4m23.092754047s) [noise=true , address=77.90.31.15:4702 , piece_id=EZ7GJ4T3OTPXVQXYNK37P64WKYNIUYL6MYZ6AJTUQBMKI26ABGNA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:30:28.949059962 (190h4m23.380140093s) [noise=true , address=77.90.31.14:4688 , piece_id=5RCAY32XV4MXIU4FYNF5GIUCDVE3NTOD3SP37KKQN4LXH5ZZKDEA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: <nil> Aug 16 15:31:19.500321044 (190h5m13.931401175s) [noise=true , address=104.36.82.59:4315 , piece_id=O6QPUXRKSQ62TR4CTNSFR6P53B6TGWQZRA6SPWXKYI3O4ANZTZ2A, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:1o5mMqSq2uht75TtSp9i4NpnkpA3AbwmNG9ffyp5zZehUqcMuz, address:104.36.82.59:4315): protocol: rpc: dial tcp 104.36.82.59:4315: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500387771 (190h5m13.931467892s) [noise=true , address=77.90.31.10:4539 , piece_id=ZBLKK5Q2WAKP62GX5OESK2GGKYXPZA6FKWB5O44E36MZC67BK22Q, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500420835 (190h5m13.931500966s) [noise=true , address=104.36.82.44:4872 , piece_id=UNJU6SUJBWGBGHQZPJYEYYENMZDJYMCOQWAFZBWMEOR6IKMKNJ7A, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: rpc: dial tcp 104.36.82.44:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500452686 (190h5m13.931532817s) [noise=true , address=104.36.82.36:4622 , piece_id=MVNGNTCN7GIJTEYHHEMQZZFYBGOESACFGQNPFJ3LN7P26QRECBYQ, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:104.36.82.36:4622): protocol: rpc: dial tcp 104.36.82.36:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500484717 (190h5m13.931564838s) [noise=true , address=104.36.82.23:4450 , piece_id=KGN27KGUY5ZQ4H5BGDJAG3HWEZJD6O7XEEQWQ6IW4OHACJEJH5ZQ, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms, address:104.36.82.23:4450): protocol: write tcp 188.166.225.170:35068->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500527738 (190h5m13.93160787s) [noise=true , address=104.36.82.34:4563 , piece_id=B56ANLGWZHTURNMPTNCFDRBAC73EJR7LEDUNWUXHMI7VZTDGHXBA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz, address:104.36.82.34:4563): protocol: rpc: dial tcp 104.36.82.34:4563: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500599526 (190h5m13.931679657s) [noise=true , address=104.36.82.60:4339 , piece_id=ZNNTY27WP7LYNREGOQWKV2QSNDZITGLD3AZLRCMUQCQPOWHFYZTA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500678517 (190h5m13.931758649s) [noise=true , address=104.36.82.38:4722 , piece_id=EHKZOC6PXHOMXCA4HG27GPFEXJBVXPPYVTRZU647MBG3UCNLYECA, seg_pos={0 0}, upload=197756]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: write tcp 188.166.225.170:49606->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:31:19.500748090 (190h5m13.931828231s) [seg_pos={0 0}, upload=197756]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:31:19.939432673 (190h5m14.370512804s) [upload=197756]: Done uploading segments... Aug 16 15:31:19.939456178 (190h5m14.370536309s) [upload=197756]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.440755766s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 7.498133665s FAIL (2025-08-16 15:30:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 726]: Uploading piece... Aug 16 15:28:27.020203021 (190h2m21.451283152s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020837635 (190h2m21.451917766s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020988324 (190h2m21.452068495s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021173278 (190h2m21.452253409s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021286745 (190h2m21.452366876s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021728570 (190h2m21.452808701s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022024927 (190h2m21.453105059s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022263874 (190h2m21.453344005s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022568146 (190h2m21.453648277s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022830999 (190h2m21.45391113s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023159919 (190h2m21.45424005s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023515449 (190h2m21.45459557s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024374543 (190h2m21.455454674s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024636975 (190h2m21.455717106s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024851636 (190h2m21.455931767s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:28.845652110 (190h2m23.276732241s) [upload=197726]: Next returned nil segment Aug 16 15:28:28.845703918 (190h2m23.276784049s) [upload=197726]: Waiting for error group managing segments... Aug 16 15:28:29.126768310 (190h2m23.557848441s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153250251 (190h2m23.584330382s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153878302 (190h2m23.584958433s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.154744730 (190h2m23.585824861s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.178067095 (190h2m23.609147226s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:35.314544157 (190h2m29.745624288s) [noise=true , address=104.36.82.13:4187 , piece_id=5X5NHWJZLY7YWKB6GPZ3EHGMDNPCORUYLZWSJ6OFGRD7JQEFKLOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:51.885334344 (190h2m46.316414475s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:52.304020451 (190h2m46.735100622s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:56.144552194 (190h2m50.575632325s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:28:56.144866246 (190h2m50.575946377s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection Aug 16 15:28:56.144907053 (190h2m50.575987184s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:28:56.144939425 (190h2m50.576019556s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection Aug 16 15:28:56.145113679 (190h2m50.57619381s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection Aug 16 15:28:56.145189925 (190h2m50.576270056s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection Aug 16 15:28:56.145271471 (190h2m50.576351632s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection Aug 16 15:28:56.145310165 (190h2m50.576390286s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection Aug 16 15:28:56.145350471 (190h2m50.576430643s) [seg_pos={0 0}, upload=197726]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:28:56.421207048 (190h2m50.852287179s) [upload=197726]: Done uploading segments... Aug 16 15:28:56.421231044 (190h2m50.852311175s) [upload=197726]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.49816218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.....X............X...X...X..................X.X..X.....X...] duration: 3.763069787s FAIL (2025-08-16 15:30:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 3.763090325s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.240301958s FAIL (2025-08-16 15:30:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.022115784 (190h3m20.453195915s) [noise=true , address=104.36.82.5:4870 , piece_id=BDTVVS4TE4VBUQY2C7KT3VKYCPPV23RG245PJ6H77GCRODG32LBA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.022479960 (190h3m20.453560092s) [noise=true , address=77.90.31.16:4753 , piece_id=DO2UWZK45U7ZJZN6FGO3U3N75SFL2LAGTSKZNM25E42BSGNOMVVQ, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.022543331 (190h3m20.453623462s) [noise=true , address=104.36.82.2:4760 , piece_id=FW5WBWO2RLKSQK7IORDZTTXXZDMSDKO2BYFU2TE66WRVM522YF4A, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.022909523 (190h3m20.453989654s) [noise=true , address=104.36.82.23:4457 , piece_id=IEAHSSVMOGUWKUBXZ653VDIUD24I5522G64M4CEB5YN3S7G7AVZQ, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.022942155 (190h3m20.454022286s) [noise=true , address=45.91.116.9:4294 , piece_id=REMHIATKU4FYSHC56X4JZYQR64JRV7EKIXC2POFRVW6DIGS45VKA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.023245756 (190h3m20.454325877s) [noise=true , address=45.91.116.8:4058 , piece_id=2VS6CNAREVLZQJ2FQINY54CJFKGIOUDCN7KLMV23JROZ2VNEO7IA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.023551551 (190h3m20.454631682s) [noise=true , address=104.36.82.59:4302 , piece_id=U2U6P3UF6JS5UJO5VRM4P36NVAP3XLTMNG47Y7DOAYXSB24TKRAA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.023890248 (190h3m20.45497038s) [noise=true , address=79.127.220.101:4061 , piece_id=6MB3ZKJBZGO6C4WVY6D57DTKWAVUJTVOTEB2EHAMRRXTWLWZTH3A, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.024333076 (190h3m20.455413207s) [noise=true , address=104.36.82.60:4339 , piece_id=U5XPHIECYMN3IKECD3KW4JDP6VI76W4YH6CF3PORVH4DYQUZ7LGQ, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.024722932 (190h3m20.455803063s) [noise=true , address=104.36.82.44:4886 , piece_id=NTFOFVRF5WINLDWTB3345XRP5KYWVOOHZCQ3FVS4EJWQ5MC5H5ZQ, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.024981857 (190h3m20.456061988s) [noise=true , address=77.90.31.21:4883 , piece_id=47DKMRQSBBINKMUPILEYBQVPTZ2T2TFE6NKJRNCNRQMXX457O6EA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.025055869 (190h3m20.456136s) [noise=true , address=77.90.31.12:4606 , piece_id=I7YAJRQ2M5TEXGQ3NH4Z4PJWN5I6KOEWUGHVSTDUAGBIMTOPGEEA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.025233739 (190h3m20.45631387s) [noise=true , address=104.36.82.38:4705 , piece_id=H755XKGU44AC4EXKNPFACWQUFKLCUI25JV52FGUMSYAQ7PXNFPWA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.025476013 (190h3m20.456556144s) [noise=true , address=79.127.201.215:4344 , piece_id=JVSQGQLQNZHIAYBCTPXRCLNDZOFBMN5UWQHH2J2LX2R2PP7B3UKA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.025713598 (190h3m20.456793729s) [noise=true , address=77.90.31.11:4563 , piece_id=JI5BERKUQRXA24BTMXV4H3B5IIEPGVIPIRPKWXZURL3DUH3MXENA, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:26.026155143 (190h3m20.457235274s) [noise=true , address=104.36.82.51:4060 , piece_id=RMSRAPNRNM3ICIEHPKJ2U6B7HS3JYA2BJFAH27SCZD2PT3R3YEGQ, seg_pos={0 0}, upload=197739]: Uploading piece... Aug 16 15:29:27.619237949 (190h3m22.05031808s) [upload=197739]: Next returned nil segment Aug 16 15:29:27.619255794 (190h3m22.050335925s) [upload=197739]: Waiting for error group managing segments... Aug 16 15:29:28.051168209 (190h3m22.48224834s) [noise=true , address=79.127.220.101:4061 , piece_id=6MB3ZKJBZGO6C4WVY6D57DTKWAVUJTVOTEB2EHAMRRXTWLWZTH3A, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:28.185835301 (190h3m22.616915432s) [noise=true , address=77.90.31.12:4606 , piece_id=I7YAJRQ2M5TEXGQ3NH4Z4PJWN5I6KOEWUGHVSTDUAGBIMTOPGEEA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:28.304672045 (190h3m22.735752166s) [noise=true , address=77.90.31.21:4883 , piece_id=47DKMRQSBBINKMUPILEYBQVPTZ2T2TFE6NKJRNCNRQMXX457O6EA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:28.601173873 (190h3m23.032254004s) [noise=true , address=77.90.31.14:4681 , piece_id=UWDS6VO6IGMDKM5NZ6BFSGIJQ74RVI6JEAPK7PL3QUIR57ROCNUQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:28.639959206 (190h3m23.071039337s) [noise=true , address=45.91.116.8:4058 , piece_id=2VS6CNAREVLZQJ2FQINY54CJFKGIOUDCN7KLMV23JROZ2VNEO7IA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:30.070914787 (190h3m24.501994918s) [noise=true , address=45.91.116.9:4294 , piece_id=REMHIATKU4FYSHC56X4JZYQR64JRV7EKIXC2POFRVW6DIGS45VKA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:29:33.637135742 (190h3m28.068215873s) [noise=true , address=79.127.201.215:4344 , piece_id=JVSQGQLQNZHIAYBCTPXRCLNDZOFBMN5UWQHH2J2LX2R2PP7B3UKA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:30:02.453586639 (190h3m56.88466678s) [noise=true , address=77.90.31.18:4836 , piece_id=IRJH5Q4I5OIZHJ4X66XS2YFCMKVGSFQRDTFTPM2V6DQGWLJGIJCA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:30:09.932090055 (190h4m4.363170176s) [noise=true , address=77.90.31.16:4753 , piece_id=DO2UWZK45U7ZJZN6FGO3U3N75SFL2LAGTSKZNM25E42BSGNOMVVQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: <nil> Aug 16 15:30:24.876960887 (190h4m19.308041008s) [noise=true , address=104.36.82.60:4339 , piece_id=U5XPHIECYMN3IKECD3KW4JDP6VI76W4YH6CF3PORVH4DYQUZ7LGQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.876997497 (190h4m19.308077628s) [noise=true , address=104.36.82.2:4760 , piece_id=FW5WBWO2RLKSQK7IORDZTTXXZDMSDKO2BYFU2TE66WRVM522YF4A, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: rpc: dial tcp 104.36.82.2:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877014018 (190h4m19.308094149s) [noise=true , address=104.36.82.44:4886 , piece_id=NTFOFVRF5WINLDWTB3345XRP5KYWVOOHZCQ3FVS4EJWQ5MC5H5ZQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877036361 (190h4m19.308116492s) [noise=true , address=104.36.82.59:4302 , piece_id=U2U6P3UF6JS5UJO5VRM4P36NVAP3XLTMNG47Y7DOAYXSB24TKRAA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:104.36.82.59:4302): protocol: rpc: dial tcp 104.36.82.59:4302: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877048034 (190h4m19.308128165s) [noise=true , address=104.36.82.51:4060 , piece_id=RMSRAPNRNM3ICIEHPKJ2U6B7HS3JYA2BJFAH27SCZD2PT3R3YEGQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:19VJKnTAges9YKYdUVoz3tb3sFU1bawxpaJ2rsBd2wmLo1zK4X, address:104.36.82.51:4060): protocol: write tcp 188.166.225.170:39994->104.36.82.51:4060: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877082018 (190h4m19.30816215s) [noise=true , address=104.36.82.5:4870 , piece_id=BDTVVS4TE4VBUQY2C7KT3VKYCPPV23RG245PJ6H77GCRODG32LBA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877087489 (190h4m19.30816762s) [noise=true , address=104.36.82.38:4705 , piece_id=H755XKGU44AC4EXKNPFACWQUFKLCUI25JV52FGUMSYAQ7PXNFPWA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:12kt12GvRGL3iNLVzYSgvs3Ha2Tzf2FApGVsK9aeuNMsQcfGUBw, address:104.36.82.38:4705): protocol: write tcp 188.166.225.170:51792->104.36.82.38:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877102658 (190h4m19.308182789s) [noise=true , address=77.90.31.11:4563 , piece_id=JI5BERKUQRXA24BTMXV4H3B5IIEPGVIPIRPKWXZURL3DUH3MXENA, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb, address:77.90.31.11:4563): protocol: rpc: dial tcp 77.90.31.11:4563: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877113539 (190h4m19.30819367s) [noise=true , address=104.36.82.23:4457 , piece_id=IEAHSSVMOGUWKUBXZ653VDIUD24I5522G64M4CEB5YN3S7G7AVZQ, seg_pos={0 0}, upload=197739]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:37320->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:24.877130892 (190h4m19.308211023s) [seg_pos={0 0}, upload=197739]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:30:25.115630789 (190h4m19.54671092s) [upload=197739]: Done uploading segments... Aug 16 15:30:25.115659003 (190h4m19.546739134s) [upload=197739]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.240313189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......X...........X......X...X.X......X..X......X..X...XX] duration: 1m0.255928216s FAIL (2025-08-16 15:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:29:18.081061850 (190h3m12.512141981s) [upload=197738]: Starting upload Aug 16 15:29:18.081120653 (190h3m12.512200784s) [upload=197738]: Done starting upload Aug 16 15:29:18.081226144 (190h3m12.512306275s) [upload=197738]: Uploading segments... Aug 16 15:29:18.081344902 (190h3m12.512425023s) [upload=197738]: Got next segment. Inline: false Aug 16 15:29:18.081395840 (190h3m12.512475971s) [upload=197738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:29:18.081401059 (190h3m12.51248119s) [upload=197738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:29:19.897530444 (190h3m14.328610575s) [seg_pos={0 0}, upload=197738]: Begin upload segment... Aug 16 15:29:19.897574969 (190h3m14.32865509s) [seg_pos={0 0}, upload=197738]: Done begin upload segment. Aug 16 15:29:19.897599366 (190h3m14.328679487s) [noise=true , address=104.36.82.39:4759 , piece_id=H5KASUOKNTMRAMCQZNWYXJKPJ5XQLFPSZTZD3TO5UI7RUPAGVGJQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.897611750 (190h3m14.328691881s) [noise=true , address=77.90.31.13:4659 , piece_id=PY7X7DMPZ55HHOJ4GCU6N2WRJYDBJW2KY7ZW6CS3P2FXJKAMZHUA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.897648780 (190h3m14.328728911s) [noise=true , address=104.36.82.59:4312 , piece_id=KHC7QW7I44RVHNAUJICUHJDMPQKQNFSXKNONNHZGYAQ2QAAELDXQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.897972560 (190h3m14.329052691s) [noise=true , address=77.90.31.12:4607 , piece_id=3R2URKIVZ6CZ4NQ26BUEUFNDT5MHGV7LHEPLBXGOC7GYJKTRBFTA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.897991507 (190h3m14.329071638s) [noise=true , address=104.36.82.5:4860 , piece_id=K3WCGLNWCXNWIWOQSUKX62DCQSAPKFOVB62CMI4VM3553IE35HKA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898267605 (190h3m14.329347756s) [noise=true , address=104.36.82.2:4760 , piece_id=RU5ZWGTXCNITZT3D7UOBKVCW2MURXTEFYGSFR5Z5K2JTQCVLI6EQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898345163 (190h3m14.329425284s) [noise=true , address=45.91.116.10:4114 , piece_id=V2H5VRSTAXGHRU3DVOR5JJPBVYOBPQELGJ6DI4OAVOYWVGMVXMAQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898583649 (190h3m14.32966378s) [noise=true , address=104.36.82.14:4229 , piece_id=EASKNCE22PEBVJ3MG5NLPZYYBU5FTJCPEQ2U43JHOHJJ5FBNA7XQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898675616 (190h3m14.329755747s) [noise=true , address=104.36.82.60:4348 , piece_id=4I7DB2CFGREGJ54TQRMIOEEOL2LDVKY73Y5IOTEUZLPLXINC65VQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898816745 (190h3m14.329896876s) [noise=true , address=104.36.82.34:4546 , piece_id=J2N5KSJU5IYBEKFLZFYJPW5ZLKVNNR4TTJG6OXDEGKBO2QVIB2GQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.898944249 (190h3m14.33002438s) [noise=true , address=79.127.219.51:4198 , piece_id=L64N5EOQBRIZ2WL2H4EHLUC4LNHFYCCXQ4QRCG4O4FC3KVMFWGBA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.899037187 (190h3m14.330117318s) [noise=true , address=104.36.82.44:4894 , piece_id=IQTVFYCW2JHX5I6GTKTDN4X2JT5AGHDE6LJLS3JOJLIKYE7GRV2A, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.899259052 (190h3m14.330339183s) [noise=true , address=79.127.201.215:4343 , piece_id=VU4WWMJTILK64MFF4S5KTKBIB5ZIQC77KASYCQAQVXBXJTSA7MZA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.899492297 (190h3m14.330572428s) [noise=true , address=104.36.82.38:4692 , piece_id=XKR2MVVFWGB6ZNX42XKGDW7SZKF6KVNLVCNCODANC76NOQST72YQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.899689876 (190h3m14.330770007s) [noise=true , address=104.36.82.23:4478 , piece_id=HBNAGYF3CMZSBHZCL7WCBTHOUESUOVF3KN2QBETVNSFCNELYOETA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.899882445 (190h3m14.330962566s) [noise=true , address=104.36.82.58:4272 , piece_id=VJGRLVKQSNCF76SGOEQVZ2YXVYMMQLN7G225C736MKPXMD4RKRNA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.900084892 (190h3m14.331165023s) [noise=true , address=77.90.31.19:4845 , piece_id=YCNBFTLSABACPS3XALKHRQCRBZ3DTYHUQKENAFWJMD5RD3PO2IHA, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:19.900274785 (190h3m14.331354906s) [noise=true , address=104.36.82.50:4046 , piece_id=GMK62F7QQJJY5WLRDMH23HZ4E7PKCMHFXOCNQJCLAHZLK2Z3V4TQ, seg_pos={0 0}, upload=197738]: Uploading piece... Aug 16 15:29:21.477482857 (190h3m15.908562988s) [upload=197738]: Next returned nil segment Aug 16 15:29:21.477501994 (190h3m15.908582125s) [upload=197738]: Waiting for error group managing segments... Aug 16 15:29:21.916666955 (190h3m16.347747086s) [noise=true , address=79.127.219.51:4198 , piece_id=L64N5EOQBRIZ2WL2H4EHLUC4LNHFYCCXQ4QRCG4O4FC3KVMFWGBA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:29:22.580538042 (190h3m17.011618173s) [noise=true , address=77.90.31.12:4607 , piece_id=3R2URKIVZ6CZ4NQ26BUEUFNDT5MHGV7LHEPLBXGOC7GYJKTRBFTA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:29:23.699372048 (190h3m18.130452179s) [noise=true , address=77.90.31.19:4845 , piece_id=YCNBFTLSABACPS3XALKHRQCRBZ3DTYHUQKENAFWJMD5RD3PO2IHA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:29:25.149675009 (190h3m19.58075514s) [noise=true , address=45.91.116.10:4114 , piece_id=V2H5VRSTAXGHRU3DVOR5JJPBVYOBPQELGJ6DI4OAVOYWVGMVXMAQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:29:25.193952129 (190h3m19.62503229s) [noise=true , address=79.127.201.215:4343 , piece_id=VU4WWMJTILK64MFF4S5KTKBIB5ZIQC77KASYCQAQVXBXJTSA7MZA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:30:05.958306678 (190h4m0.389386809s) [noise=true , address=104.36.82.50:4046 , piece_id=GMK62F7QQJJY5WLRDMH23HZ4E7PKCMHFXOCNQJCLAHZLK2Z3V4TQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: <nil> Aug 16 15:30:17.155283580 (190h4m11.586363721s) [noise=true , address=104.36.82.14:4229 , piece_id=EASKNCE22PEBVJ3MG5NLPZYYBU5FTJCPEQ2U43JHOHJJ5FBNA7XQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12uiCMvJ7pnwXfN57CVu8zR73tikwUZiXr3sG7Xotd9w6KVnViL, address:104.36.82.14:4229): protocol: storage node overloaded, request limit: 2000 Aug 16 15:30:17.440559669 (190h4m11.87163979s) [noise=true , address=104.36.82.59:4312 , piece_id=KHC7QW7I44RVHNAUJICUHJDMPQKQNFSXKNONNHZGYAQ2QAAELDXQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY, address:104.36.82.59:4312): protocol: write tcp 188.166.225.170:52362->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440606989 (190h4m11.87168712s) [noise=true , address=104.36.82.2:4760 , piece_id=RU5ZWGTXCNITZT3D7UOBKVCW2MURXTEFYGSFR5Z5K2JTQCVLI6EQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: rpc: dial tcp 104.36.82.2:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440642708 (190h4m11.871722839s) [noise=true , address=104.36.82.60:4348 , piece_id=4I7DB2CFGREGJ54TQRMIOEEOL2LDVKY73Y5IOTEUZLPLXINC65VQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: rpc: dial tcp 104.36.82.60:4348: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440756937 (190h4m11.871837068s) [noise=true , address=104.36.82.44:4894 , piece_id=IQTVFYCW2JHX5I6GTKTDN4X2JT5AGHDE6LJLS3JOJLIKYE7GRV2A, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440811681 (190h4m11.871891812s) [noise=true , address=104.36.82.38:4692 , piece_id=XKR2MVVFWGB6ZNX42XKGDW7SZKF6KVNLVCNCODANC76NOQST72YQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:104.36.82.38:4692): protocol: write tcp 188.166.225.170:45168->104.36.82.38:4692: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440848491 (190h4m11.871928622s) [noise=true , address=104.36.82.5:4860 , piece_id=K3WCGLNWCXNWIWOQSUKX62DCQSAPKFOVB62CMI4VM3553IE35HKA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440879691 (190h4m11.871959822s) [noise=true , address=104.36.82.34:4546 , piece_id=J2N5KSJU5IYBEKFLZFYJPW5ZLKVNNR4TTJG6OXDEGKBO2QVIB2GQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:104.36.82.34:4546): protocol: rpc: dial tcp 104.36.82.34:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440953893 (190h4m11.872034014s) [noise=true , address=77.90.31.13:4659 , piece_id=PY7X7DMPZ55HHOJ4GCU6N2WRJYDBJW2KY7ZW6CS3P2FXJKAMZHUA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: rpc: dial tcp 77.90.31.13:4659: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.440993909 (190h4m11.87207404s) [noise=true , address=104.36.82.58:4272 , piece_id=VJGRLVKQSNCF76SGOEQVZ2YXVYMMQLN7G225C736MKPXMD4RKRNA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:1yY39UpswsTLoWz4353JykK35fg8KB4AJWPatKGFgqTt4gDMcR, address:104.36.82.58:4272): protocol: rpc: dial tcp 104.36.82.58:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.441021111 (190h4m11.872101232s) [noise=true , address=104.36.82.39:4759 , piece_id=H5KASUOKNTMRAMCQZNWYXJKPJ5XQLFPSZTZD3TO5UI7RUPAGVGJQ, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12AEfTC9phpBk8uTbN6k7oDvz6svgwdPH1PaU8jfuuBxhHTRtTR, address:104.36.82.39:4759): protocol: rpc: dial tcp 104.36.82.39:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.441045889 (190h4m11.87212602s) [noise=true , address=104.36.82.23:4478 , piece_id=HBNAGYF3CMZSBHZCL7WCBTHOUESUOVF3KN2QBETVNSFCNELYOETA, seg_pos={0 0}, upload=197738]: Done uploading piece. err: ecclient: upload failed (node:12EvdsFq3xEc1mmD5y1PMsNo685MbFj1zCMby8WBvxBee4emtpR, address:104.36.82.23:4478): protocol: write tcp 188.166.225.170:46494->104.36.82.23:4478: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:30:17.441074714 (190h4m11.872154845s) [seg_pos={0 0}, upload=197738]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 15:30:17.694549364 (190h4m12.125629546s) [upload=197738]: Done uploading segments... Aug 16 15:30:17.694594561 (190h4m12.125674742s) [upload=197738]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.255945939s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 25.625990487s FAIL (2025-08-16 15:30:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 726]: Uploading piece... Aug 16 15:28:27.020203021 (190h2m21.451283152s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020837635 (190h2m21.451917766s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020988324 (190h2m21.452068495s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021173278 (190h2m21.452253409s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021286745 (190h2m21.452366876s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021728570 (190h2m21.452808701s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022024927 (190h2m21.453105059s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022263874 (190h2m21.453344005s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022568146 (190h2m21.453648277s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022830999 (190h2m21.45391113s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023159919 (190h2m21.45424005s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023515449 (190h2m21.45459557s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024374543 (190h2m21.455454674s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024636975 (190h2m21.455717106s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024851636 (190h2m21.455931767s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:28.845652110 (190h2m23.276732241s) [upload=197726]: Next returned nil segment Aug 16 15:28:28.845703918 (190h2m23.276784049s) [upload=197726]: Waiting for error group managing segments... Aug 16 15:28:29.126768310 (190h2m23.557848441s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153250251 (190h2m23.584330382s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153878302 (190h2m23.584958433s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.154744730 (190h2m23.585824861s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.178067095 (190h2m23.609147226s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:35.314544157 (190h2m29.745624288s) [noise=true , address=104.36.82.13:4187 , piece_id=5X5NHWJZLY7YWKB6GPZ3EHGMDNPCORUYLZWSJ6OFGRD7JQEFKLOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:51.885334344 (190h2m46.316414475s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:52.304020451 (190h2m46.735100622s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:56.144552194 (190h2m50.575632325s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:28:56.144866246 (190h2m50.575946377s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection Aug 16 15:28:56.144907053 (190h2m50.575987184s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:28:56.144939425 (190h2m50.576019556s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection Aug 16 15:28:56.145113679 (190h2m50.57619381s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection Aug 16 15:28:56.145189925 (190h2m50.576270056s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection Aug 16 15:28:56.145271471 (190h2m50.576351632s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection Aug 16 15:28:56.145310165 (190h2m50.576390286s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection Aug 16 15:28:56.145350471 (190h2m50.576430643s) [seg_pos={0 0}, upload=197726]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:28:56.421207048 (190h2m50.852287179s) [upload=197726]: Done uploading segments... Aug 16 15:28:56.421231044 (190h2m50.852311175s) [upload=197726]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 25.626015094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [X.....X............X...X...X..................X.X..X.....X..] duration: 6.199887914s FAIL (2025-08-16 15:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 6.199935816s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.234346711s FAIL (2025-08-16 15:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ss=77.90.31.15:4707 , piece_id=PEATCZIT76WEJPILBRUCJ3VZLAEC65AKNOWOWYS526OKJXMA63HA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.569765062 (190h2m20.000845183s) [noise=true , address=104.36.82.60:4338 , piece_id=62BKRC26TULUTAD4LEBQUEAKQRNRN4FM3WRMGXDX773JLTG36WFA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.570153996 (190h2m20.001234117s) [noise=true , address=104.36.82.4:4807 , piece_id=UFSJ4P7HZ2F65TLG5GVKORYSCQOXJKKDDKNVNEN2OW4CKCFQQFPQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.571597839 (190h2m20.00267797s) [noise=true , address=104.36.82.44:4890 , piece_id=U2JEVH2W3QZ75RVGGVY3J4XFSBA3BG2AVT4R3UXDRTP3DOCB26MA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.571916448 (190h2m20.002996579s) [noise=true , address=104.36.82.36:4629 , piece_id=JOQ5BGDDOKDUUWKMFUEF5DD7JKEBD6VDGMURLKINFHFPEM54U4RQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.572203397 (190h2m20.003283518s) [noise=true , address=77.90.31.13:4658 , piece_id=JAPYG2ZWPZPLC5XHRD6SEQGN6NWI6VPDROFJPULSCWFEMZWD4TNQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.572268371 (190h2m20.003348522s) [noise=true , address=77.90.31.16:4761 , piece_id=UR2TPXOXEYMJOX5C53CFCLKBOYIOHSDZBPQR4KNS2D7XXGCBXSPA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.572568886 (190h2m20.003649017s) [noise=true , address=77.90.31.19:4846 , piece_id=4YZCND5PWPVVZJ5Y4C7AYPVNB3HXDPMVKGO6YOFNA23CKRXFWNMQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.572587102 (190h2m20.003667233s) [noise=true , address=77.90.31.14:4664 , piece_id=YDJRIEZFSD6HKP2FY6ERFNIUG527KIWYAQLOEZ7ITDM6KRFINCZA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.572907154 (190h2m20.003987285s) [noise=true , address=45.91.116.11:4125 , piece_id=THTSHJ4F7G4Y5ADU277QQE2OE7RF5ODGAHFQNOTGCURAPKHYDWVQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573152233 (190h2m20.004232354s) [noise=true , address=104.36.82.59:4315 , piece_id=7ETCZQVMHEEW5H74LRMOKP4TJZZAOJT2WNX5LWYXYOPPPPJFDEEA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573321387 (190h2m20.004401508s) [noise=true , address=104.36.82.61:4385 , piece_id=YHOYQDS4FMIHK7RY2HRVLE63I6UF3XH5QXBL3AC2QZCOWPMEG7HQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573394606 (190h2m20.004474737s) [noise=true , address=104.36.82.5:4862 , piece_id=YANPJ7ZMVBASGMVMCJRPAXN7CKHNXLWVXAVMNUID3YNCEN5ENC3Q, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573639545 (190h2m20.004719686s) [noise=true , address=77.90.31.18:4827 , piece_id=IWJLIIZI3RLDRADM6RWPHUS3ZJKIV7STRVJURT6KM3D7RFM6U3LA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573703378 (190h2m20.004783509s) [noise=true , address=66.23.193.175:4556 , piece_id=M2VLQZKCEDUSCMAADVDOO7GFUCH3C4QKLN4OGQMLFVCKHMV7JT7A, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.573948086 (190h2m20.005028217s) [noise=true , address=104.36.82.23:4454 , piece_id=SOHHRHXRRD4SIOO5R2ZHQZZNUKGQEWN6UQJ552XNRNVA5NYPMREQ, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.574207492 (190h2m20.005287613s) [noise=true , address=77.90.31.10:4527 , piece_id=II2ZMQMGLTUMAOUWM2A6QUHAMXKFP4NQXMLBZ7E2QS5XCSDOA6EA, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:25.574461177 (190h2m20.005541308s) [noise=true , address=104.36.82.17:4359 , piece_id=3JBS4P4VH6SNEAEHU6SSEICIUNYNBL4NR642PV3LCV2HAQHVBM7A, seg_pos={0 0}, upload=197725]: Uploading piece... Aug 16 15:28:26.726260679 (190h2m21.15734081s) [upload=197725]: Next returned nil segment Aug 16 15:28:26.726276791 (190h2m21.157356922s) [upload=197725]: Waiting for error group managing segments... Aug 16 15:28:27.706078471 (190h2m22.137158602s) [noise=true , address=104.36.82.17:4359 , piece_id=3JBS4P4VH6SNEAEHU6SSEICIUNYNBL4NR642PV3LCV2HAQHVBM7A, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:28:27.790751174 (190h2m22.221831295s) [noise=true , address=66.23.193.175:4556 , piece_id=M2VLQZKCEDUSCMAADVDOO7GFUCH3C4QKLN4OGQMLFVCKHMV7JT7A, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:28:30.139658631 (190h2m24.570738762s) [noise=true , address=104.36.82.61:4385 , piece_id=YHOYQDS4FMIHK7RY2HRVLE63I6UF3XH5QXBL3AC2QZCOWPMEG7HQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:28:36.079307852 (190h2m30.510387973s) [noise=true , address=77.90.31.18:4827 , piece_id=IWJLIIZI3RLDRADM6RWPHUS3ZJKIV7STRVJURT6KM3D7RFM6U3LA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:28:41.649770437 (190h2m36.080850558s) [noise=true , address=77.90.31.13:4658 , piece_id=JAPYG2ZWPZPLC5XHRD6SEQGN6NWI6VPDROFJPULSCWFEMZWD4TNQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:28:42.093480736 (190h2m36.524560867s) [noise=true , address=45.91.116.11:4125 , piece_id=THTSHJ4F7G4Y5ADU277QQE2OE7RF5ODGAHFQNOTGCURAPKHYDWVQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:29:00.107988323 (190h2m54.539068454s) [noise=true , address=77.90.31.14:4664 , piece_id=YDJRIEZFSD6HKP2FY6ERFNIUG527KIWYAQLOEZ7ITDM6KRFINCZA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:29:05.447576772 (190h2m59.878656904s) [noise=true , address=77.90.31.15:4707 , piece_id=PEATCZIT76WEJPILBRUCJ3VZLAEC65AKNOWOWYS526OKJXMA63HA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:29:12.175812110 (190h3m6.606892241s) [noise=true , address=77.90.31.16:4761 , piece_id=UR2TPXOXEYMJOX5C53CFCLKBOYIOHSDZBPQR4KNS2D7XXGCBXSPA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: <nil> Aug 16 15:29:24.414059466 (190h3m18.845139597s) [noise=true , address=77.90.31.10:4527 , piece_id=II2ZMQMGLTUMAOUWM2A6QUHAMXKFP4NQXMLBZ7E2QS5XCSDOA6EA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2, address:77.90.31.10:4527): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:29:24.415148290 (190h3m18.846228421s) [noise=true , address=77.90.31.19:4846 , piece_id=4YZCND5PWPVVZJ5Y4C7AYPVNB3HXDPMVKGO6YOFNA23CKRXFWNMQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: rpc: dial tcp 77.90.31.19:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:24.415187394 (190h3m18.846267525s) [noise=true , address=104.36.82.44:4890 , piece_id=U2JEVH2W3QZ75RVGGVY3J4XFSBA3BG2AVT4R3UXDRTP3DOCB26MA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:24.415216700 (190h3m18.846296821s) [noise=true , address=104.36.82.36:4629 , piece_id=JOQ5BGDDOKDUUWKMFUEF5DD7JKEBD6VDGMURLKINFHFPEM54U4RQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: rpc: dial tcp 104.36.82.36:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:24.415245245 (190h3m18.846325376s) [noise=true , address=104.36.82.4:4807 , piece_id=UFSJ4P7HZ2F65TLG5GVKORYSCQOXJKKDDKNVNEN2OW4CKCFQQFPQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:104.36.82.4:4807): protocol: rpc: dial tcp 104.36.82.4:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:24.415275923 (190h3m18.846356054s) [noise=true , address=104.36.82.5:4862 , piece_id=YANPJ7ZMVBASGMVMCJRPAXN7CKHNXLWVXAVMNUID3YNCEN5ENC3Q, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:12sUT7LBYFqWMUhx7MMjEmyiFDnbJnjSVndNbTaHioaVLLSXYue, address:104.36.82.5:4862): protocol: read tcp 188.166.225.170:37710->104.36.82.5:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:24.415345627 (190h3m18.846425758s) [noise=true , address=104.36.82.60:4338 , piece_id=62BKRC26TULUTAD4LEBQUEAKQRNRN4FM3WRMGXDX773JLTG36WFA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:104.36.82.60:4338): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:29:24.415443754 (190h3m18.846523885s) [noise=true , address=104.36.82.23:4454 , piece_id=SOHHRHXRRD4SIOO5R2ZHQZZNUKGQEWN6UQJ552XNRNVA5NYPMREQ, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2, address:104.36.82.23:4454): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:29:24.415590886 (190h3m18.846671007s) [noise=true , address=104.36.82.59:4315 , piece_id=7ETCZQVMHEEW5H74LRMOKP4TJZZAOJT2WNX5LWYXYOPPPPJFDEEA, seg_pos={0 0}, upload=197725]: Done uploading piece. err: ecclient: upload failed (node:1o5mMqSq2uht75TtSp9i4NpnkpA3AbwmNG9ffyp5zZehUqcMuz, address:104.36.82.59:4315): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:29:24.415621725 (190h3m18.846701846s) [seg_pos={0 0}, upload=197725]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:29:24.646981711 (190h3m19.078061842s) [upload=197725]: Done uploading segments... Aug 16 15:29:24.647004725 (190h3m19.078084856s) [upload=197725]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.234373173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.......X...........X......X...X.X......X..X......X..X...X] duration: 1m0.466361239s FAIL (2025-08-16 15:29:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Aug 16 15:28:13.007386100 (190h2m7.438466231s) [upload=197722]: Starting upload Aug 16 15:28:13.007452256 (190h2m7.438532387s) [upload=197722]: Done starting upload Aug 16 15:28:13.007657690 (190h2m7.438737821s) [upload=197722]: Uploading segments... Aug 16 15:28:13.007781155 (190h2m7.438861286s) [upload=197722]: Got next segment. Inline: false Aug 16 15:28:13.007815281 (190h2m7.438895402s) [upload=197722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:13.007820281 (190h2m7.438900412s) [upload=197722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:15.565274593 (190h2m9.996354734s) [seg_pos={0 0}, upload=197722]: Begin upload segment... Aug 16 15:28:15.565336111 (190h2m9.996416252s) [seg_pos={0 0}, upload=197722]: Done begin upload segment. Aug 16 15:28:15.565348244 (190h2m9.996428425s) [noise=true , address=77.90.31.15:4711 , piece_id=7U5TAIM7IAYBPRMPRFWNLQ7UAP5S6PSFO76GAEHXXCEUZ3V3KLRA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.565361369 (190h2m9.99644151s) [noise=true , address=104.36.82.44:4874 , piece_id=VWLKCUAIXNYQMIZ6UXMIYFA32LCP6RU3FOOYQTDMZXXY45I67XRQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.565477661 (190h2m9.996557823s) [noise=true , address=77.90.31.19:4845 , piece_id=JPLBGSHIEP4XQ475C5WPRYETS2HROKBAC6QOB62AZUL2CS6IG4MQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.566224029 (190h2m9.99730415s) [noise=true , address=104.36.82.23:4451 , piece_id=6R7KTCSGUBHTZVFOV6PKK4P7CIMCLU3ZYJRAWEMO77PRC57GLOPQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.566259608 (190h2m9.997339759s) [noise=true , address=104.36.82.36:4630 , piece_id=KOIKPBAHOS574BJUQTCGIYMTYK6HJRZ7VWNESENXY4GQ2RILULDQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.566866820 (190h2m9.997946951s) [noise=true , address=77.90.31.10:4532 , piece_id=4UKYDQB52WNNYGYOMPW4F3653BOCQGFUM77W7LMXP2B5OTFUXO3A, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.566876638 (190h2m9.997956779s) [noise=true , address=79.127.220.101:4061 , piece_id=K5D3ZA64P2EFSJY57KMLAIHC3IJEBSIS5FTYB64R36YVZGNMEPIA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.567301231 (190h2m9.998381362s) [noise=true , address=45.91.116.11:4135 , piece_id=3K5JNBTST2I3ZTIUOTHL4UMQAB7N223JRDLW3XZHEWCNXUYWS5VQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.567309477 (190h2m9.998389618s) [noise=true , address=104.36.82.59:4334 , piece_id=M3IPN6WE2GN7FHTO6EDILLKCLNK3ZBLCSFNGG4LJGDTYEPM7LAGQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.567344684 (190h2m9.998424835s) [noise=true , address=104.36.82.39:4753 , piece_id=SMLP4EJDIIU7JNPZ2HFKRFBHIE4KJ3CLGZSKUOYBVICLS53IEGSQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.567740281 (190h2m9.998820412s) [noise=true , address=77.90.31.18:4808 , piece_id=7JIG5H7UUCE226ACIBPBIRJXXKISPCD4ACM4AR47TO5CWPYR7BHA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.567744760 (190h2m9.998824891s) [noise=true , address=45.91.116.7:4013 , piece_id=G4KVQCTI6T2CY2ANF6K5DZG6O3TJFHOLRBC47GG4UXY6IXSIA36A, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.568140256 (190h2m9.999220387s) [noise=true , address=45.91.116.5:4301 , piece_id=GTLWITWA7EAKFXNHNMUDOLWQHBZ423H6SNF24ETFYDLHP7TNHH2A, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.568145496 (190h2m9.999225627s) [noise=true , address=77.90.31.21:4885 , piece_id=L3MPJRHYEJW5O6ZDFXTGKEZSQFBOXJEIADFUP2SRN2BS7CQ3G5VA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.570046824 (190h2m10.001126955s) [noise=true , address=104.36.82.60:4362 , piece_id=YPMSV6FKKGSR3PJWC7Z3NGRRILYNESUD5ZRPPFG74YEASRQJ2EUA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.570533356 (190h2m10.001613487s) [noise=true , address=79.127.201.215:4353 , piece_id=S7CQMFCAV6KSAYAJOPC5PTCLZQFBDDBVKUBCU6TJBY7VLNORMCHQ, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.571471490 (190h2m10.002551611s) [noise=true , address=104.36.82.35:4595 , piece_id=2HP5PSR7SQNNW5KCCEOJJ5Y7EUQBXGXBOF2PF7LYECUMOXIALQVA, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:15.571868991 (190h2m10.002949122s) [noise=true , address=66.23.193.175:4557 , piece_id=H6R2APA2P2COHZLXSSL2U2WJERSW6BGUWSQGKQE7V2YOUZ3KFK7A, seg_pos={0 0}, upload=197722]: Uploading piece... Aug 16 15:28:16.755520990 (190h2m11.186601111s) [upload=197722]: Next returned nil segment Aug 16 15:28:16.755536831 (190h2m11.186616962s) [upload=197722]: Waiting for error group managing segments... Aug 16 15:28:17.424886098 (190h2m11.855966229s) [noise=true , address=79.127.220.101:4061 , piece_id=K5D3ZA64P2EFSJY57KMLAIHC3IJEBSIS5FTYB64R36YVZGNMEPIA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:17.558918436 (190h2m11.989998557s) [noise=true , address=79.127.201.215:4353 , piece_id=S7CQMFCAV6KSAYAJOPC5PTCLZQFBDDBVKUBCU6TJBY7VLNORMCHQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:18.327786234 (190h2m12.758866395s) [noise=true , address=104.36.82.44:4874 , piece_id=VWLKCUAIXNYQMIZ6UXMIYFA32LCP6RU3FOOYQTDMZXXY45I67XRQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:18.910600345 (190h2m13.341680476s) [noise=true , address=104.36.82.60:4362 , piece_id=YPMSV6FKKGSR3PJWC7Z3NGRRILYNESUD5ZRPPFG74YEASRQJ2EUA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:21.133264688 (190h2m15.564344809s) [noise=true , address=45.91.116.5:4301 , piece_id=GTLWITWA7EAKFXNHNMUDOLWQHBZ423H6SNF24ETFYDLHP7TNHH2A, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:23.198475969 (190h2m17.629556101s) [noise=true , address=77.90.31.18:4808 , piece_id=7JIG5H7UUCE226ACIBPBIRJXXKISPCD4ACM4AR47TO5CWPYR7BHA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:23.677393764 (190h2m18.108473895s) [noise=true , address=77.90.31.15:4711 , piece_id=7U5TAIM7IAYBPRMPRFWNLQ7UAP5S6PSFO76GAEHXXCEUZ3V3KLRA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:24.304022640 (190h2m18.735102771s) [noise=true , address=45.91.116.7:4013 , piece_id=G4KVQCTI6T2CY2ANF6K5DZG6O3TJFHOLRBC47GG4UXY6IXSIA36A, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:26.780039831 (190h2m21.211119962s) [noise=true , address=77.90.31.21:4885 , piece_id=L3MPJRHYEJW5O6ZDFXTGKEZSQFBOXJEIADFUP2SRN2BS7CQ3G5VA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:28.937698061 (190h2m23.368778192s) [noise=true , address=66.23.193.175:4557 , piece_id=H6R2APA2P2COHZLXSSL2U2WJERSW6BGUWSQGKQE7V2YOUZ3KFK7A, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:32.682641084 (190h2m27.113721215s) [noise=true , address=77.90.31.10:4532 , piece_id=4UKYDQB52WNNYGYOMPW4F3653BOCQGFUM77W7LMXP2B5OTFUXO3A, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:28:57.123459674 (190h2m51.554539805s) [noise=true , address=45.91.116.11:4135 , piece_id=3K5JNBTST2I3ZTIUOTHL4UMQAB7N223JRDLW3XZHEWCNXUYWS5VQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: <nil> Aug 16 15:29:12.363097123 (190h3m6.794177254s) [noise=true , address=104.36.82.23:4451 , piece_id=6R7KTCSGUBHTZVFOV6PKK4P7CIMCLU3ZYJRAWEMO77PRC57GLOPQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: rpc: dial tcp 104.36.82.23:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363139844 (190h3m6.794219975s) [noise=true , address=104.36.82.39:4753 , piece_id=SMLP4EJDIIU7JNPZ2HFKRFBHIE4KJ3CLGZSKUOYBVICLS53IEGSQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: write tcp 188.166.225.170:46412->104.36.82.39:4753: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363162618 (190h3m6.794242749s) [noise=true , address=104.36.82.59:4334 , piece_id=M3IPN6WE2GN7FHTO6EDILLKCLNK3ZBLCSFNGG4LJGDTYEPM7LAGQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:12MaiBbEtrHEaXrMy5hhbYDQsGfTCZuT6C62aFxUnXPE5vRW3kg, address:104.36.82.59:4334): protocol: write tcp 188.166.225.170:58032->104.36.82.59:4334: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363193677 (190h3m6.794273798s) [noise=true , address=77.90.31.19:4845 , piece_id=JPLBGSHIEP4XQ475C5WPRYETS2HROKBAC6QOB62AZUL2CS6IG4MQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: write tcp 188.166.225.170:39510->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363218164 (190h3m6.794298295s) [noise=true , address=104.36.82.35:4595 , piece_id=2HP5PSR7SQNNW5KCCEOJJ5Y7EUQBXGXBOF2PF7LYECUMOXIALQVA, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:12amZR7XxhSNjYTK414HmFCTkMz6DDhT8AjvS1Xcg5C9nzvEK8d, address:104.36.82.35:4595): protocol: rpc: dial tcp 104.36.82.35:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363233554 (190h3m6.794313685s) [noise=true , address=104.36.82.36:4630 , piece_id=KOIKPBAHOS574BJUQTCGIYMTYK6HJRZ7VWNESENXY4GQ2RILULDQ, seg_pos={0 0}, upload=197722]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: rpc: dial tcp 104.36.82.36:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:29:12.363251719 (190h3m6.79433185s) [seg_pos={0 0}, upload=197722]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:29:12.827529922 (190h3m7.258610053s) [upload=197722]: Done uploading segments... Aug 16 15:29:12.827556653 (190h3m7.258636784s) [upload=197722]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.466381297s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 30.278332093s FAIL (2025-08-16 15:28:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 726]: Uploading piece... Aug 16 15:28:27.020203021 (190h2m21.451283152s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020837635 (190h2m21.451917766s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.020988324 (190h2m21.452068495s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021173278 (190h2m21.452253409s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021286745 (190h2m21.452366876s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.021728570 (190h2m21.452808701s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022024927 (190h2m21.453105059s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022263874 (190h2m21.453344005s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022568146 (190h2m21.453648277s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.022830999 (190h2m21.45391113s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023159919 (190h2m21.45424005s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.023515449 (190h2m21.45459557s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024374543 (190h2m21.455454674s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024636975 (190h2m21.455717106s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:27.024851636 (190h2m21.455931767s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Uploading piece... Aug 16 15:28:28.845652110 (190h2m23.276732241s) [upload=197726]: Next returned nil segment Aug 16 15:28:28.845703918 (190h2m23.276784049s) [upload=197726]: Waiting for error group managing segments... Aug 16 15:28:29.126768310 (190h2m23.557848441s) [noise=true , address=104.36.82.7:4910 , piece_id=2QCKONXCJP7ZYMZ2FXSA4HOHO4CIH5K7XQ7VDT4IHY346AWQ3YDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153250251 (190h2m23.584330382s) [noise=true , address=104.36.82.10:4078 , piece_id=3CRL5YWHPMUKVSKXETTKTZO7MGYN7R5B236XMYPWRJGEFYAMCA6Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.153878302 (190h2m23.584958433s) [noise=true , address=104.36.82.11:4114 , piece_id=OOXNQ7VYCYOMEXCWM6GSJ24YUS5UW6FTF6C5B7WSPTKMAQMOBKDA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.154744730 (190h2m23.585824861s) [noise=true , address=104.36.82.6:4873 , piece_id=JV5GJWDNI4CPXFRKUCDF3W7KWMSBHHBDXT6NEUEYYK5ZUGJKUS3Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:29.178067095 (190h2m23.609147226s) [noise=true , address=104.36.82.15:4257 , piece_id=VZANVPOC2J3NJUVV5GNBZI2L7JTPWXRVIBGVWEA3XMNMSLUDKHBA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:35.314544157 (190h2m29.745624288s) [noise=true , address=104.36.82.13:4187 , piece_id=5X5NHWJZLY7YWKB6GPZ3EHGMDNPCORUYLZWSJ6OFGRD7JQEFKLOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:51.885334344 (190h2m46.316414475s) [noise=true , address=77.90.31.10:4520 , piece_id=ZN4FAHZYJSKS26JI3B3QSU6P7LJBRN5FWVTRPQQGI7GBLYJT4LSA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:52.304020451 (190h2m46.735100622s) [noise=true , address=45.91.116.14:4191 , piece_id=DC5JKTLWOOYEXQXC3V67IJU4B3NKRDDIEERH4HJUNDTJEGYPNYEQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: <nil> Aug 16 15:28:56.144552194 (190h2m50.575632325s) [noise=true , address=45.91.116.13:4155 , piece_id=E5IXEU5TSTFAQYVST5ZSWQDTKLOK2KYNVO2KA6Q6BV6QJYBTE64A, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:28:56.144866246 (190h2m50.575946377s) [noise=true , address=77.90.31.21:4878 , piece_id=S4QSPN7XTWUQVTIWIHXYVN2N5KYG2GPWSVDR2DR6A7RU6Y3XS3OA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42392->77.90.31.21:4878: use of closed network connection Aug 16 15:28:56.144907053 (190h2m50.575987184s) [noise=true , address=104.36.82.9:4006 , piece_id=EPFABTKH5E4YAGB6YWMKNUQFHPKHGGY7AWQ45Y43IEJDOBMX3ODA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:28:56.144939425 (190h2m50.576019556s) [noise=true , address=77.90.31.18:4805 , piece_id=N63QQEEIGVAX5PMMNLIGDKAGUZRYP4JF533RT2T6ZGHNW7ERDG5Q, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54304->77.90.31.18:4805: use of closed network connection Aug 16 15:28:56.145113679 (190h2m50.57619381s) [noise=true , address=45.91.116.5:4298 , piece_id=EIDWBVE23QLWTBQPYPMZIPDK5OM6UMHIE5QPRD3HAQT7SCHEHWOQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50184->45.91.116.5:4298: use of closed network connection Aug 16 15:28:56.145189925 (190h2m50.576270056s) [noise=true , address=45.91.116.11:4118 , piece_id=E5EGT5E75XFCVK5JGAQ2MGNVP6QSKKX2LWSOFYVZYO2YGAFUSSAQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54708->45.91.116.11:4118: use of closed network connection Aug 16 15:28:56.145271471 (190h2m50.576351632s) [noise=true , address=77.90.31.10:4519 , piece_id=OFB26E47WK5LR5ZNJGLZHV3JWVGKCMWIF7CYYN3RQEBRS42S3VOA, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55306->77.90.31.10:4519: use of closed network connection Aug 16 15:28:56.145310165 (190h2m50.576390286s) [noise=true , address=45.91.116.14:4189 , piece_id=S7QRFM6CYWF2BJHSMWFFHE4LVIZR3MWHONM6UVLPROXS6XKUTUTQ, seg_pos={0 0}, upload=197726]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50588->45.91.116.14:4189: use of closed network connection Aug 16 15:28:56.145350471 (190h2m50.576430643s) [seg_pos={0 0}, upload=197726]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:28:56.421207048 (190h2m50.852287179s) [upload=197726]: Done uploading segments... Aug 16 15:28:56.421231044 (190h2m50.852311175s) [upload=197726]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.278346781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [.X.....X............X...X...X..................X.X..X.....X.] duration: 9.409162163s FAIL (2025-08-16 15:28:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 9.409180618s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [..X.....X............X...X...X..................X.X..X.....X] duration: 29.610500469s FAIL (2025-08-16 15:28:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 29.610525216s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.......X...........X......X...X.X......X..X......X..X...] duration: 17.025052907s FAIL (2025-08-16 15:28:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0934b95503ad489b687c81e4da4873e8b18cdfa218bd6f26a2bcf3aa39b6798d != before 20b81e033bae496bc2b3ad61754df60c4333151ef3d4cd038739f60c89db5160 ======LOGS====== Aug 16 15:26:22.446202381 (190h0m16.877282512s) [upload=197691]: Starting upload Aug 16 15:26:22.446251054 (190h0m16.877331185s) [upload=197691]: Done starting upload Aug 16 15:26:22.446390641 (190h0m16.877470772s) [upload=197691]: Uploading segments... Aug 16 15:26:22.446650007 (190h0m16.877730138s) [upload=197691]: Got next segment. Inline: false Aug 16 15:26:22.446691527 (190h0m16.877771658s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:26:22.446697017 (190h0m16.877777138s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:26:23.257729163 (190h0m17.688809294s) [seg_pos={0 0}, upload=197691]: Begin upload segment... Aug 16 15:26:23.257775001 (190h0m17.688855122s) [seg_pos={0 0}, upload=197691]: Done begin upload segment. Aug 16 15:26:23.257814025 (190h0m17.688894156s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.257900009 (190h0m17.68898014s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258223278 (190h0m17.689303399s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258612503 (190h0m17.689692634s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258634555 (190h0m17.689714686s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258985426 (190h0m17.690065548s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259107269 (190h0m17.6901874s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259409497 (190h0m17.690489628s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259554094 (190h0m17.690634215s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259729109 (190h0m17.69080924s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259864909 (190h0m17.69094504s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260002402 (190h0m17.691082533s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260160796 (190h0m17.691240927s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260269684 (190h0m17.691349805s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260514582 (190h0m17.691594713s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260621256 (190h0m17.691701387s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261111314 (190h0m17.692191435s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261849416 (190h0m17.692929537s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:24.382815872 (190h0m18.813896003s) [upload=197691]: Next returned nil segment Aug 16 15:26:24.382831352 (190h0m18.813911483s) [upload=197691]: Waiting for error group managing segments... Aug 16 15:26:24.861555466 (190h0m19.292635597s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.230475851 (190h0m19.661555982s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.319976643 (190h0m19.751056764s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.346387928 (190h0m19.777468059s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.356810711 (190h0m19.787890842s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.600010590 (190h0m21.031090721s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.714588524 (190h0m21.145668666s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.824996800 (190h0m21.256076931s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.575986420 (190h0m22.007066551s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.655302203 (190h0m22.086382334s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.125026257 (190h0m22.556106388s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.143110480 (190h0m22.574190611s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:29.052829122 (190h0m23.483909253s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262903348 (190h0m24.693983479s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262932874 (190h0m24.694013015s) [seg_pos={0 0}, upload=197691]: Segment reached optimal threshold of 14 pieces. Aug 16 15:26:30.263338199 (190h0m24.69441833s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ys2boqq8tJDTRRSGV47s1uhZqA68LzZuQ3HwpKwghUPc5Dwy7): protocol: expected piece hash; context canceled; EOF Aug 16 15:26:30.263496973 (190h0m24.694577104s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: rpc: dial tcp 104.36.82.58:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263593468 (190h0m24.694673599s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: rpc: dial tcp 104.36.82.60:4349: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263638073 (190h0m24.694718204s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF): protocol: write tcp 188.166.225.170:48080->104.36.82.23:4470: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263660316 (190h0m24.694740447s) [seg_pos={0 0}, upload=197691]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:26:30.263725691 (190h0m24.694805822s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:26:30.263730140 (190h0m24.694810271s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:26:42.386392675 (190h0m36.817472806s) [upload=197691]: Done uploading segments... Aug 16 15:26:42.386413875 (190h0m36.817494036s) [upload=197691]: Upload finished. err: <nil> Duration: 17.025070221s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 13.011231691s FAIL (2025-08-16 15:28:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.011260435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...X.......X...........X......X...X.X......X..X......X..X..] duration: 25.683569649s FAIL (2025-08-16 15:27:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0934b95503ad489b687c81e4da4873e8b18cdfa218bd6f26a2bcf3aa39b6798d != before 20b81e033bae496bc2b3ad61754df60c4333151ef3d4cd038739f60c89db5160 ======LOGS====== Aug 16 15:26:22.446202381 (190h0m16.877282512s) [upload=197691]: Starting upload Aug 16 15:26:22.446251054 (190h0m16.877331185s) [upload=197691]: Done starting upload Aug 16 15:26:22.446390641 (190h0m16.877470772s) [upload=197691]: Uploading segments... Aug 16 15:26:22.446650007 (190h0m16.877730138s) [upload=197691]: Got next segment. Inline: false Aug 16 15:26:22.446691527 (190h0m16.877771658s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:26:22.446697017 (190h0m16.877777138s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:26:23.257729163 (190h0m17.688809294s) [seg_pos={0 0}, upload=197691]: Begin upload segment... Aug 16 15:26:23.257775001 (190h0m17.688855122s) [seg_pos={0 0}, upload=197691]: Done begin upload segment. Aug 16 15:26:23.257814025 (190h0m17.688894156s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.257900009 (190h0m17.68898014s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258223278 (190h0m17.689303399s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258612503 (190h0m17.689692634s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258634555 (190h0m17.689714686s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258985426 (190h0m17.690065548s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259107269 (190h0m17.6901874s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259409497 (190h0m17.690489628s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259554094 (190h0m17.690634215s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259729109 (190h0m17.69080924s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259864909 (190h0m17.69094504s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260002402 (190h0m17.691082533s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260160796 (190h0m17.691240927s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260269684 (190h0m17.691349805s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260514582 (190h0m17.691594713s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260621256 (190h0m17.691701387s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261111314 (190h0m17.692191435s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261849416 (190h0m17.692929537s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:24.382815872 (190h0m18.813896003s) [upload=197691]: Next returned nil segment Aug 16 15:26:24.382831352 (190h0m18.813911483s) [upload=197691]: Waiting for error group managing segments... Aug 16 15:26:24.861555466 (190h0m19.292635597s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.230475851 (190h0m19.661555982s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.319976643 (190h0m19.751056764s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.346387928 (190h0m19.777468059s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.356810711 (190h0m19.787890842s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.600010590 (190h0m21.031090721s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.714588524 (190h0m21.145668666s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.824996800 (190h0m21.256076931s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.575986420 (190h0m22.007066551s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.655302203 (190h0m22.086382334s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.125026257 (190h0m22.556106388s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.143110480 (190h0m22.574190611s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:29.052829122 (190h0m23.483909253s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262903348 (190h0m24.693983479s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262932874 (190h0m24.694013015s) [seg_pos={0 0}, upload=197691]: Segment reached optimal threshold of 14 pieces. Aug 16 15:26:30.263338199 (190h0m24.69441833s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ys2boqq8tJDTRRSGV47s1uhZqA68LzZuQ3HwpKwghUPc5Dwy7): protocol: expected piece hash; context canceled; EOF Aug 16 15:26:30.263496973 (190h0m24.694577104s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: rpc: dial tcp 104.36.82.58:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263593468 (190h0m24.694673599s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: rpc: dial tcp 104.36.82.60:4349: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263638073 (190h0m24.694718204s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF): protocol: write tcp 188.166.225.170:48080->104.36.82.23:4470: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263660316 (190h0m24.694740447s) [seg_pos={0 0}, upload=197691]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:26:30.263725691 (190h0m24.694805822s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:26:30.263730140 (190h0m24.694810271s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:26:42.386392675 (190h0m36.817472806s) [upload=197691]: Done uploading segments... Aug 16 15:26:42.386413875 (190h0m36.817494036s) [upload=197691]: Upload finished. err: <nil> Duration: 25.683661035s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.040173014s FAIL (2025-08-16 15:27:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.040197922s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 6.234032397s FAIL (2025-08-16 15:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.234054028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [...X.....X............X...X...X..................X.X..X.....] duration: 26.937937897s FAIL (2025-08-16 15:27:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 26.937968305s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.340998944s FAIL (2025-08-16 15:27:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.341023682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX...X.......X...........X......X...X.X......X..X......X..X.] duration: 19.926880977s FAIL (2025-08-16 15:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0934b95503ad489b687c81e4da4873e8b18cdfa218bd6f26a2bcf3aa39b6798d != before 20b81e033bae496bc2b3ad61754df60c4333151ef3d4cd038739f60c89db5160 ======LOGS====== Aug 16 15:26:22.446202381 (190h0m16.877282512s) [upload=197691]: Starting upload Aug 16 15:26:22.446251054 (190h0m16.877331185s) [upload=197691]: Done starting upload Aug 16 15:26:22.446390641 (190h0m16.877470772s) [upload=197691]: Uploading segments... Aug 16 15:26:22.446650007 (190h0m16.877730138s) [upload=197691]: Got next segment. Inline: false Aug 16 15:26:22.446691527 (190h0m16.877771658s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:26:22.446697017 (190h0m16.877777138s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:26:23.257729163 (190h0m17.688809294s) [seg_pos={0 0}, upload=197691]: Begin upload segment... Aug 16 15:26:23.257775001 (190h0m17.688855122s) [seg_pos={0 0}, upload=197691]: Done begin upload segment. Aug 16 15:26:23.257814025 (190h0m17.688894156s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.257900009 (190h0m17.68898014s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258223278 (190h0m17.689303399s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258612503 (190h0m17.689692634s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258634555 (190h0m17.689714686s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258985426 (190h0m17.690065548s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259107269 (190h0m17.6901874s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259409497 (190h0m17.690489628s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259554094 (190h0m17.690634215s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259729109 (190h0m17.69080924s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259864909 (190h0m17.69094504s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260002402 (190h0m17.691082533s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260160796 (190h0m17.691240927s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260269684 (190h0m17.691349805s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260514582 (190h0m17.691594713s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260621256 (190h0m17.691701387s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261111314 (190h0m17.692191435s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261849416 (190h0m17.692929537s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:24.382815872 (190h0m18.813896003s) [upload=197691]: Next returned nil segment Aug 16 15:26:24.382831352 (190h0m18.813911483s) [upload=197691]: Waiting for error group managing segments... Aug 16 15:26:24.861555466 (190h0m19.292635597s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.230475851 (190h0m19.661555982s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.319976643 (190h0m19.751056764s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.346387928 (190h0m19.777468059s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.356810711 (190h0m19.787890842s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.600010590 (190h0m21.031090721s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.714588524 (190h0m21.145668666s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.824996800 (190h0m21.256076931s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.575986420 (190h0m22.007066551s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.655302203 (190h0m22.086382334s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.125026257 (190h0m22.556106388s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.143110480 (190h0m22.574190611s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:29.052829122 (190h0m23.483909253s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262903348 (190h0m24.693983479s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262932874 (190h0m24.694013015s) [seg_pos={0 0}, upload=197691]: Segment reached optimal threshold of 14 pieces. Aug 16 15:26:30.263338199 (190h0m24.69441833s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ys2boqq8tJDTRRSGV47s1uhZqA68LzZuQ3HwpKwghUPc5Dwy7): protocol: expected piece hash; context canceled; EOF Aug 16 15:26:30.263496973 (190h0m24.694577104s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: rpc: dial tcp 104.36.82.58:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263593468 (190h0m24.694673599s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: rpc: dial tcp 104.36.82.60:4349: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263638073 (190h0m24.694718204s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF): protocol: write tcp 188.166.225.170:48080->104.36.82.23:4470: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263660316 (190h0m24.694740447s) [seg_pos={0 0}, upload=197691]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:26:30.263725691 (190h0m24.694805822s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:26:30.263730140 (190h0m24.694810271s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:26:42.386392675 (190h0m36.817472806s) [upload=197691]: Done uploading segments... Aug 16 15:26:42.386413875 (190h0m36.817494036s) [upload=197691]: Upload finished. err: <nil> Duration: 19.926903601s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 6.532849765s FAIL (2025-08-16 15:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.532872798s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 9.091529832s FAIL (2025-08-16 15:26:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.091559029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX...X.......X...........X......X...X.X......X..X......X..X] duration: 23.849492774s FAIL (2025-08-16 15:26:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0934b95503ad489b687c81e4da4873e8b18cdfa218bd6f26a2bcf3aa39b6798d != before 20b81e033bae496bc2b3ad61754df60c4333151ef3d4cd038739f60c89db5160 ======LOGS====== Aug 16 15:26:22.446202381 (190h0m16.877282512s) [upload=197691]: Starting upload Aug 16 15:26:22.446251054 (190h0m16.877331185s) [upload=197691]: Done starting upload Aug 16 15:26:22.446390641 (190h0m16.877470772s) [upload=197691]: Uploading segments... Aug 16 15:26:22.446650007 (190h0m16.877730138s) [upload=197691]: Got next segment. Inline: false Aug 16 15:26:22.446691527 (190h0m16.877771658s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:26:22.446697017 (190h0m16.877777138s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:26:23.257729163 (190h0m17.688809294s) [seg_pos={0 0}, upload=197691]: Begin upload segment... Aug 16 15:26:23.257775001 (190h0m17.688855122s) [seg_pos={0 0}, upload=197691]: Done begin upload segment. Aug 16 15:26:23.257814025 (190h0m17.688894156s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.257900009 (190h0m17.68898014s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258223278 (190h0m17.689303399s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258612503 (190h0m17.689692634s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258634555 (190h0m17.689714686s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.258985426 (190h0m17.690065548s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259107269 (190h0m17.6901874s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259409497 (190h0m17.690489628s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259554094 (190h0m17.690634215s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259729109 (190h0m17.69080924s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.259864909 (190h0m17.69094504s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260002402 (190h0m17.691082533s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260160796 (190h0m17.691240927s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260269684 (190h0m17.691349805s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260514582 (190h0m17.691594713s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.260621256 (190h0m17.691701387s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261111314 (190h0m17.692191435s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:23.261849416 (190h0m17.692929537s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Uploading piece... Aug 16 15:26:24.382815872 (190h0m18.813896003s) [upload=197691]: Next returned nil segment Aug 16 15:26:24.382831352 (190h0m18.813911483s) [upload=197691]: Waiting for error group managing segments... Aug 16 15:26:24.861555466 (190h0m19.292635597s) [noise=true , address=66.23.193.175:4554 , piece_id=W5X3K5S2UP7GXW6NJPAWYCHCW6HX2QYVL6YUQVG474XAWRORWFOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.230475851 (190h0m19.661555982s) [noise=true , address=104.36.82.2:4759 , piece_id=PT2BPKP2VJMQ2FVSNVPG6C4Y5EE5ZHN7XFKKX7CLL67RRPA4MFRA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.319976643 (190h0m19.751056764s) [noise=true , address=104.36.82.36:4625 , piece_id=UGAORRE6GA5ZDCDLM7HUWJN6ISHMOJW33Z27U7W7CTZTVDJUCOGQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.346387928 (190h0m19.777468059s) [noise=true , address=79.127.201.214:4320 , piece_id=CMANF2BNKET4KAWSWM5ZIVVPM2FQQAT5CPI72NX6NVESZFNIIXPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:25.356810711 (190h0m19.787890842s) [noise=true , address=104.36.82.38:4707 , piece_id=27WQPHXBJM3O54T554433N5SAAPYVVCZI7DSYA2MYYIBYEXOEUBQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.600010590 (190h0m21.031090721s) [noise=true , address=45.91.116.4:4964 , piece_id=FOPNS4IA5SUT734ZCYRSK5WHGOAHQYX654XOY4JLSE3T2MB7WIJQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.714588524 (190h0m21.145668666s) [noise=true , address=79.127.201.215:4347 , piece_id=VOLCFTFSCMLNNGOBXQXXHUSRU3JC5JI7F5S773SJJWFO5RYXKSRQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:26.824996800 (190h0m21.256076931s) [noise=true , address=45.91.116.11:4125 , piece_id=UD6HFUTZNC4HXKFSXZCOFSU4A4IL6OERKSAU2PT6BFTD4TCX6KVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.575986420 (190h0m22.007066551s) [noise=true , address=77.90.31.11:4578 , piece_id=2FPR7BTG6LDBJXICHVDH25Z4D5C7MMGQ5UI2NTR7GJ2SWSNFMQOQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:27.655302203 (190h0m22.086382334s) [noise=true , address=45.91.116.8:4077 , piece_id=PGF62AXGZT7KTCZLO3UT3B7MVGGWQWZ42IJUKKN34WKNVLKYDM3A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.125026257 (190h0m22.556106388s) [noise=true , address=104.36.82.44:4877 , piece_id=FIEFSH5WKZTVZYKT4Z4QQPB6NRBNJL7JJ4XNJLXJ7L43MCJSUBVQ, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:28.143110480 (190h0m22.574190611s) [noise=true , address=77.90.31.15:4718 , piece_id=RYY6GK3DRPQWNSRKWCEPCZT4WFDO5VGCV53X4J5VSMZ54OIELU6Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:29.052829122 (190h0m23.483909253s) [noise=true , address=77.90.31.14:4681 , piece_id=BJY4UL4YPYHBQECM2DIHTLCYRKDZ7LPXEUJ6UDHVBCWWWIIPF23Q, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262903348 (190h0m24.693983479s) [noise=true , address=104.36.82.39:4728 , piece_id=KLETA2ORCEMQ4BQPIDKUNQH27E4DRRHUZ4UYPQWOTS6OGGV23JOA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: <nil> Aug 16 15:26:30.262932874 (190h0m24.694013015s) [seg_pos={0 0}, upload=197691]: Segment reached optimal threshold of 14 pieces. Aug 16 15:26:30.263338199 (190h0m24.69441833s) [noise=true , address=104.36.82.59:4326 , piece_id=XBP33FXBGLASF2MRTJVT4PYA2DDDQDGZQ75V5O3NWGTQQTO7RU5A, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ys2boqq8tJDTRRSGV47s1uhZqA68LzZuQ3HwpKwghUPc5Dwy7): protocol: expected piece hash; context canceled; EOF Aug 16 15:26:30.263496973 (190h0m24.694577104s) [noise=true , address=104.36.82.58:4270 , piece_id=K5WW4E2A7AQADVERTVDTVKVDCJZHSBYUNOUU2MOSBZ3O46Y3GYPA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: rpc: dial tcp 104.36.82.58:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263593468 (190h0m24.694673599s) [noise=true , address=104.36.82.60:4349 , piece_id=2XLBVYRG6YPNPGB5PUDSCVTX5ABPQUU2Z3C2EW5HX4IAJ4EA4HEA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: rpc: dial tcp 104.36.82.60:4349: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263638073 (190h0m24.694718204s) [noise=true , address=104.36.82.23:4470 , piece_id=RRZH5IAZBMQC46EICSNJXCN7B4CRKX4ZY4LMZDAHCSGNFBNZQXLA, seg_pos={0 0}, upload=197691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF): protocol: write tcp 188.166.225.170:48080->104.36.82.23:4470: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:26:30.263660316 (190h0m24.694740447s) [seg_pos={0 0}, upload=197691]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:26:30.263725691 (190h0m24.694805822s) [upload=197691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:26:30.263730140 (190h0m24.694810271s) [upload=197691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:26:42.386392675 (190h0m36.817472806s) [upload=197691]: Done uploading segments... Aug 16 15:26:42.386413875 (190h0m36.817494036s) [upload=197691]: Upload finished. err: <nil> Duration: 23.84951129s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 14.706993998s FAIL (2025-08-16 15:26:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.707033534s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 449.426326ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:26:22 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 449.455001ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXX...X.......X...........X......X...X.X......X..X......X..] duration: 32.337430657s FAIL (2025-08-16 15:26:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:24:16.043180969 (189h58m10.474261101s) [upload=197652]: Starting upload Aug 16 15:24:16.043232859 (189h58m10.47431299s) [upload=197652]: Done starting upload Aug 16 15:24:16.043434676 (189h58m10.474514807s) [upload=197652]: Uploading segments... Aug 16 15:24:16.043547622 (189h58m10.474627753s) [upload=197652]: Got next segment. Inline: false Aug 16 15:24:16.044202655 (189h58m10.475282786s) [upload=197652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:16.044210340 (189h58m10.475290471s) [upload=197652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:33.584642687 (189h58m28.015722818s) [seg_pos={0 0}, upload=197652]: Begin upload segment... Aug 16 15:24:33.584685658 (189h58m28.01576579s) [seg_pos={0 0}, upload=197652]: Done begin upload segment. Aug 16 15:24:33.584702882 (189h58m28.015783013s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.584734041 (189h58m28.015814172s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585089772 (189h58m28.016169903s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585940379 (189h58m28.017020511s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586214754 (189h58m28.017294876s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586499039 (189h58m28.01757916s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586713138 (189h58m28.017793269s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586942437 (189h58m28.018022568s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587144163 (189h58m28.018224284s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587349707 (189h58m28.018429838s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587546774 (189h58m28.018626895s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588622011 (189h58m28.019702132s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588854114 (189h58m28.019934245s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589101488 (189h58m28.020181619s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589320447 (189h58m28.020400568s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589534476 (189h58m28.020614597s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589725933 (189h58m28.020806064s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589853377 (189h58m28.020933508s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:34.815342067 (189h58m29.246422198s) [upload=197652]: Next returned nil segment Aug 16 15:24:34.815359119 (189h58m29.24643924s) [upload=197652]: Waiting for error group managing segments... Aug 16 15:24:35.129613185 (189h58m29.560693316s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.594697010 (189h58m30.025777141s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.608397716 (189h58m30.039477837s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.660219792 (189h58m30.091299923s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:39.049878441 (189h58m33.480958572s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:45.026661192 (189h58m39.457741323s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:47.496042626 (189h58m41.927122757s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:51.102194976 (189h58m45.533275107s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:52.685339387 (189h58m47.116419518s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:56.496926897 (189h58m50.928007028s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:58.193475055 (189h58m52.624555186s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.414748015 (189h58m59.845828146s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.497516565 (189h58m59.928596686s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:15.398949383 (189h59m9.830029514s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8, address:104.36.82.58:4280): protocol: rpc: dial tcp 104.36.82.58:4280: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.398978479 (189h59m9.83005861s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:15.398989299 (189h59m9.83006943s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399057961 (189h59m9.830138092s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17EmBobdmk16xm9girzx3asYwkhdCKKZCCPrBLwWTS4ujMdGmi, address:104.36.82.60:4357): protocol: write tcp 188.166.225.170:33440->104.36.82.60:4357: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399107014 (189h59m9.830187135s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: write tcp 188.166.225.170:38582->104.36.82.44:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:25:15.399125410 (189h59m9.830205541s) [seg_pos={0 0}, upload=197652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:25:16.066732284 (189h59m10.497812425s) [upload=197652]: Done uploading segments... Aug 16 15:25:16.066761029 (189h59m10.49784117s) [upload=197652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.337449052s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 849.623636ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:26:21 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 849.647983ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 738.453513ms FAIL (2025-08-16 15:26:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 738.477689ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [....X.....X............X...X...X..................X.X..X....] duration: 13.249164323s FAIL (2025-08-16 15:25:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 13.24918332s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 5.83381646s FAIL (2025-08-16 15:25:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.833836607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.640381091s FAIL (2025-08-16 15:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.640399336s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 910.61625ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:25:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 910.639004ms edge gateway vhost us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)