Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 11.742912587s FAIL (2025-08-16 15:45:27 UTC) orbiter warning: context deadline exceeded Duration: 11.74293494s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 22.770791372s FAIL (2025-08-16 15:45:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.770803375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000645105s FAIL (2025-08-16 15:45:14 UTC) orbiter warning: context deadline exceeded Duration: 30.000671948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...X...X..................X.X..X.....X......................] duration: 2.437806117s FAIL (2025-08-16 15:45:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.437827349s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..................................X......X..XXXXXXXXXX.XX.XX] duration: 30.230535915s FAIL (2025-08-16 15:45:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:44:38.231371664 (190h18m32.662451795s) [upload=197992]: Starting upload Aug 16 15:44:38.231461156 (190h18m32.662541297s) [upload=197992]: Done starting upload Aug 16 15:44:38.231579953 (190h18m32.662660084s) [upload=197992]: Uploading segments... Aug 16 15:44:38.231882412 (190h18m32.662962543s) [upload=197992]: Got next segment. Inline: false Aug 16 15:44:38.231926707 (190h18m32.663006838s) [upload=197992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:44:38.231931807 (190h18m32.663011938s) [upload=197992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:44:38.486431123 (190h18m32.917511264s) [seg_pos={0 0}, upload=197992]: Begin upload segment... Aug 16 15:44:38.486491789 (190h18m32.91757192s) [seg_pos={0 0}, upload=197992]: Done begin upload segment. Aug 16 15:44:38.486519041 (190h18m32.917599172s) [noise=true , address=77.90.31.21:4877 , piece_id=YYK6FDXTROJ5XULXFCOQPYZCCEA42G5VFUYJTSP2DOYBLTCO7N5A, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.486980234 (190h18m32.918060365s) [noise=true , address=77.90.31.11:4555 , piece_id=YJJOE2DGFWLB6W5462HRXMDIGSU74JPJRPJALP7RLXW74CJFLE2Q, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.487069715 (190h18m32.918149836s) [noise=true , address=104.36.82.10:4079 , piece_id=W2SARWJPJB42CIEUQD2Y4WXWRJVWKKO53SRK6VY5SJO7EZ2UKAWA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.487370560 (190h18m32.918450702s) [noise=true , address=104.36.82.15:4259 , piece_id=FVME2L4SM4HUD5K2OU4JAXIPAU6QU2XIU33JOYVTMLF6ZBOAXQGQ, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.487469469 (190h18m32.9185496s) [noise=true , address=104.36.82.14:4223 , piece_id=ISU25A4YRMWAUQHXRSMQNCLWMPBPP4TRU4DKHJ5TNJC775LLAT3A, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.487744125 (190h18m32.918824256s) [noise=true , address=77.90.31.13:4628 , piece_id=LQ23VCZI74SSNLLBUFC44CBSX3XN6BZ7DN5GMKHJOU2WFN7VUPIQ, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.487833256 (190h18m32.918913377s) [noise=true , address=45.91.116.5:4297 , piece_id=3TSYWWW4YK2ABZ7QSZQIS6DHLSV74RZS3WMQVFSAUO7XFVLZ4HIA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488074408 (190h18m32.919154539s) [noise=true , address=45.91.116.8:4048 , piece_id=LQLW4ENHBWBZUNIGVKXXWHJSSRAHEEZVL4S4MKOLTYLNSTOWUEJA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488094285 (190h18m32.919174416s) [noise=true , address=104.36.82.9:4006 , piece_id=7KQS2NZGVMPODWDA5NZGFCOOMF3BACXQ54FODD4HRKU3RGUO3MFQ, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488438604 (190h18m32.919518735s) [noise=true , address=45.91.116.10:4082 , piece_id=FMACJMG3ZPTQ5NGNWT25JZT7UOU2HP7HIJDKZKAVX32ZMRWTFCPA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488499510 (190h18m32.919579631s) [noise=true , address=77.90.31.19:4841 , piece_id=UV2QYVM2KBCOXHLWXTNT26W4BWZXFLAJG3LRMTNIGLMXCKHB6NKA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488760621 (190h18m32.919840752s) [noise=true , address=45.91.116.14:4191 , piece_id=Q3HPXQB2JQ56MIR3LE7PAFKFGMZHPIKCV7ZW3ATSL4HJAYQYSCWA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.488785487 (190h18m32.919865618s) [noise=true , address=104.36.82.12:4151 , piece_id=B64WMII7VKOT37IJITLJ3KQL7IAWBXA2LJDGWVJYH75YPSSMIX6A, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.489068640 (190h18m32.920148771s) [noise=true , address=104.36.82.7:4912 , piece_id=ZIZRJE5XF2UUQHRYM5JV4HEPESTMV7Q3GV4U3ZDDZT5EYBOTPOXA, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.489369395 (190h18m32.920449526s) [noise=true , address=104.36.82.11:4115 , piece_id=KLBHH47QPX3SFPXUNSFM5SNAWBKPP2BT24MOQ4H6N57WVLGF4EYQ, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:38.489607310 (190h18m32.920687441s) [noise=true , address=45.91.116.5:4298 , piece_id=AUSIRPO3K2Z4JW54RJQCM322H4MWNLQEOE5COXLNQE3FGAIPYQTQ, seg_pos={0 0}, upload=197992]: Uploading piece... Aug 16 15:44:40.179638710 (190h18m34.610718841s) [upload=197992]: Next returned nil segment Aug 16 15:44:40.179655894 (190h18m34.610736025s) [upload=197992]: Waiting for error group managing segments... Aug 16 15:44:40.522025670 (190h18m34.953105811s) [noise=true , address=45.91.116.5:4298 , piece_id=AUSIRPO3K2Z4JW54RJQCM322H4MWNLQEOE5COXLNQE3FGAIPYQTQ, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.532375013 (190h18m34.963455164s) [noise=true , address=104.36.82.15:4259 , piece_id=FVME2L4SM4HUD5K2OU4JAXIPAU6QU2XIU33JOYVTMLF6ZBOAXQGQ, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.543837295 (190h18m34.974917426s) [noise=true , address=104.36.82.7:4912 , piece_id=ZIZRJE5XF2UUQHRYM5JV4HEPESTMV7Q3GV4U3ZDDZT5EYBOTPOXA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.616510638 (190h18m35.047590759s) [noise=true , address=77.90.31.11:4555 , piece_id=YJJOE2DGFWLB6W5462HRXMDIGSU74JPJRPJALP7RLXW74CJFLE2Q, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.625511749 (190h18m35.05659187s) [noise=true , address=77.90.31.21:4877 , piece_id=YYK6FDXTROJ5XULXFCOQPYZCCEA42G5VFUYJTSP2DOYBLTCO7N5A, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.688313604 (190h18m35.119393735s) [noise=true , address=45.91.116.5:4297 , piece_id=3TSYWWW4YK2ABZ7QSZQIS6DHLSV74RZS3WMQVFSAUO7XFVLZ4HIA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.698555682 (190h18m35.129635813s) [noise=true , address=104.36.82.12:4151 , piece_id=B64WMII7VKOT37IJITLJ3KQL7IAWBXA2LJDGWVJYH75YPSSMIX6A, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:40.765415570 (190h18m35.196495691s) [noise=true , address=45.91.116.8:4048 , piece_id=LQLW4ENHBWBZUNIGVKXXWHJSSRAHEEZVL4S4MKOLTYLNSTOWUEJA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:41.033356908 (190h18m35.464437039s) [noise=true , address=45.91.116.14:4191 , piece_id=Q3HPXQB2JQ56MIR3LE7PAFKFGMZHPIKCV7ZW3ATSL4HJAYQYSCWA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:41.046817744 (190h18m35.477897875s) [noise=true , address=104.36.82.10:4079 , piece_id=W2SARWJPJB42CIEUQD2Y4WXWRJVWKKO53SRK6VY5SJO7EZ2UKAWA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:41.203382854 (190h18m35.634462985s) [noise=true , address=45.91.116.10:4082 , piece_id=FMACJMG3ZPTQ5NGNWT25JZT7UOU2HP7HIJDKZKAVX32ZMRWTFCPA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:44:50.853514515 (190h18m45.284594636s) [noise=true , address=77.90.31.13:4628 , piece_id=LQ23VCZI74SSNLLBUFC44CBSX3XN6BZ7DN5GMKHJOU2WFN7VUPIQ, seg_pos={0 0}, upload=197992]: Done uploading piece. err: <nil> Aug 16 15:45:07.580060042 (190h19m2.011140173s) [noise=true , address=77.90.31.19:4841 , piece_id=UV2QYVM2KBCOXHLWXTNT26W4BWZXFLAJG3LRMTNIGLMXCKHB6NKA, seg_pos={0 0}, upload=197992]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:07.580079269 (190h19m2.01115939s) [noise=true , address=104.36.82.11:4115 , piece_id=KLBHH47QPX3SFPXUNSFM5SNAWBKPP2BT24MOQ4H6N57WVLGF4EYQ, seg_pos={0 0}, upload=197992]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:45:07.580110588 (190h19m2.01119072s) [noise=true , address=104.36.82.14:4223 , piece_id=ISU25A4YRMWAUQHXRSMQNCLWMPBPP4TRU4DKHJ5TNJC775LLAT3A, seg_pos={0 0}, upload=197992]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:37998->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:07.580189409 (190h19m2.01126954s) [noise=true , address=104.36.82.9:4006 , piece_id=7KQS2NZGVMPODWDA5NZGFCOOMF3BACXQ54FODD4HRKU3RGUO3MFQ, seg_pos={0 0}, upload=197992]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38584->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:07.580210109 (190h19m2.01129024s) [seg_pos={0 0}, upload=197992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:45:07.809465694 (190h19m2.240545825s) [upload=197992]: Done uploading segments... Aug 16 15:45:07.809489128 (190h19m2.240569259s) [upload=197992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230554862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXX] duration: 1m0.344299539s FAIL (2025-08-16 15:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:44:01.009972132 (190h17m55.441052263s) [upload=197977]: Starting upload Aug 16 15:44:01.010026546 (190h17m55.441106677s) [upload=197977]: Done starting upload Aug 16 15:44:01.010148981 (190h17m55.441229102s) [upload=197977]: Uploading segments... Aug 16 15:44:01.010280151 (190h17m55.441360282s) [upload=197977]: Got next segment. Inline: false Aug 16 15:44:01.010327191 (190h17m55.441407322s) [upload=197977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:44:01.010331510 (190h17m55.441411631s) [upload=197977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:44:01.537872682 (190h17m55.968952813s) [seg_pos={0 0}, upload=197977]: Begin upload segment... Aug 16 15:44:01.538004905 (190h17m55.969085046s) [seg_pos={0 0}, upload=197977]: Done begin upload segment. Aug 16 15:44:01.538019694 (190h17m55.969099825s) [noise=true , address=104.36.82.5:4868 , piece_id=OTY3Z37532MJFIZK4OHIS6WJP7HGJEEDVRMI7QQYQK55SSNEB6TA, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.538062946 (190h17m55.969143077s) [noise=true , address=45.91.116.14:4207 , piece_id=YWPC5D4GBWAGP56PWJZ6UNMQTIYCCPADHQ24AWVMOA6UHC33QHXA, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.538600625 (190h17m55.969680756s) [noise=true , address=156.146.43.225:4457 , piece_id=KZOFFMZSRI3ZG2S6BNJPAFDHAYWQGW4X57YLFWOU73K3VGUPK7MA, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.539037121 (190h17m55.970117262s) [noise=true , address=104.36.82.9:4030 , piece_id=PX3CBZZUG7XQGH77N4LSSZG75GTC2FGRI5AUYQVX4YHLM2CPEOSQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.539447255 (190h17m55.970527386s) [noise=true , address=77.90.31.16:4767 , piece_id=BOORRAOVBCCNLMXXZTT2W4FU52QC5RDMXBTCVKYEXUPFNCTM3QTQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.539857490 (190h17m55.970937621s) [noise=true , address=77.90.31.16:4764 , piece_id=AGCZXPRENGTWZ634TII4YDLFWGSNIE6NCIN4JUIE7ZBWNTVGQGYQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.540220134 (190h17m55.971300265s) [noise=true , address=104.36.82.60:4364 , piece_id=JVMUDVXMWJPAPPLIA4LX2FUYBEMU5IUGG3SM3ACIUTVXR4OM25AA, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.540594080 (190h17m55.971674211s) [noise=true , address=77.90.31.18:4830 , piece_id=JG6STQ5ZKVC4TABGDBAKW2PHLQ75PXNPR3FDYDJFSCRT4MEZ3LRQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.540941714 (190h17m55.972021845s) [noise=true , address=66.23.193.175:4555 , piece_id=4STI6LROZJYHVG3SNK27SE3Y3P5PKARI3WD3FZNC2KJ5NUAQAD6A, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.541388831 (190h17m55.972468972s) [noise=true , address=45.91.116.13:4178 , piece_id=P2DBPCICAQ2MRDKEEOX54R4NDGKDHR44DVCQXQPP5KAP64B7IY6A, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.541714222 (190h17m55.972794353s) [noise=true , address=104.36.82.44:4902 , piece_id=JWTN6VMW6BOCJ755D46MJDHHOI5OX5R2GGRC6EVLMWZFSBRW4SLQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.542168943 (190h17m55.973249074s) [noise=true , address=104.36.82.4:4835 , piece_id=IPH2MRFOUNVFIA45XR4PR5P5VSMOR2WCLCFHOSBRES2M73SVE6CQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.542527619 (190h17m55.97360775s) [noise=true , address=77.90.31.12:4601 , piece_id=3QC6A6AYG5XIJA7JQETSKT7LZXESTSH6BM5DWZ7OFCEOI3GMIU2Q, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.542918597 (190h17m55.973998728s) [noise=true , address=77.90.31.21:4885 , piece_id=54FL7RKLRW2F2HB42JUEKZAP27NAMDSQDNUUH7JTCRWJK36LI6FQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.543250131 (190h17m55.974330262s) [noise=true , address=77.90.31.11:4580 , piece_id=MYYAQMCHLGV5HX57V44NZHHDNI4RWHGRL6WHHYXQ7RC7GJVNG3SA, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.543579842 (190h17m55.974659973s) [noise=true , address=79.127.220.101:4071 , piece_id=VXKRP6H37IDHOT6BWFQQR5RR7RGH6ZUJNLBDSLLE63VPYH3V4T5A, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.543974448 (190h17m55.975054579s) [noise=true , address=45.91.116.12:4247 , piece_id=DZYLNNR4ZGYJUL52G2KQDOFY5GTSDE223T2CXI7NIJWJT77MGSDQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:01.544330589 (190h17m55.97541071s) [noise=true , address=79.127.201.215:4354 , piece_id=IS3JFY6UKQBOXMITGVOZFL3XHD76KLIHRSEC2TAIAJVABZJKCZBQ, seg_pos={0 0}, upload=197977]: Uploading piece... Aug 16 15:44:02.729593259 (190h17m57.160673391s) [upload=197977]: Next returned nil segment Aug 16 15:44:02.729610112 (190h17m57.160690243s) [upload=197977]: Waiting for error group managing segments... Aug 16 15:44:03.642966700 (190h17m58.074046831s) [noise=true , address=77.90.31.11:4580 , piece_id=MYYAQMCHLGV5HX57V44NZHHDNI4RWHGRL6WHHYXQ7RC7GJVNG3SA, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:03.775956291 (190h17m58.207036422s) [noise=true , address=77.90.31.16:4764 , piece_id=AGCZXPRENGTWZ634TII4YDLFWGSNIE6NCIN4JUIE7ZBWNTVGQGYQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:03.830162077 (190h17m58.261242198s) [noise=true , address=77.90.31.12:4601 , piece_id=3QC6A6AYG5XIJA7JQETSKT7LZXESTSH6BM5DWZ7OFCEOI3GMIU2Q, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:03.920530148 (190h17m58.351610269s) [noise=true , address=45.91.116.14:4207 , piece_id=YWPC5D4GBWAGP56PWJZ6UNMQTIYCCPADHQ24AWVMOA6UHC33QHXA, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:03.950919755 (190h17m58.381999886s) [noise=true , address=77.90.31.16:4767 , piece_id=BOORRAOVBCCNLMXXZTT2W4FU52QC5RDMXBTCVKYEXUPFNCTM3QTQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:04.228769951 (190h17m58.659850082s) [noise=true , address=156.146.43.225:4457 , piece_id=KZOFFMZSRI3ZG2S6BNJPAFDHAYWQGW4X57YLFWOU73K3VGUPK7MA, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:05.415143780 (190h17m59.846223911s) [noise=true , address=77.90.31.21:4885 , piece_id=54FL7RKLRW2F2HB42JUEKZAP27NAMDSQDNUUH7JTCRWJK36LI6FQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:05.589945843 (190h18m0.021025964s) [noise=true , address=79.127.220.101:4071 , piece_id=VXKRP6H37IDHOT6BWFQQR5RR7RGH6ZUJNLBDSLLE63VPYH3V4T5A, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:06.642621093 (190h18m1.073701224s) [noise=true , address=79.127.201.215:4354 , piece_id=IS3JFY6UKQBOXMITGVOZFL3XHD76KLIHRSEC2TAIAJVABZJKCZBQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:06.931449565 (190h18m1.362529696s) [noise=true , address=66.23.193.175:4555 , piece_id=4STI6LROZJYHVG3SNK27SE3Y3P5PKARI3WD3FZNC2KJ5NUAQAD6A, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:07.133378804 (190h18m1.564458925s) [noise=true , address=45.91.116.12:4247 , piece_id=DZYLNNR4ZGYJUL52G2KQDOFY5GTSDE223T2CXI7NIJWJT77MGSDQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:11.240926919 (190h18m5.67200705s) [noise=true , address=77.90.31.18:4830 , piece_id=JG6STQ5ZKVC4TABGDBAKW2PHLQ75PXNPR3FDYDJFSCRT4MEZ3LRQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:44:16.009062183 (190h18m10.440142314s) [noise=true , address=45.91.116.13:4178 , piece_id=P2DBPCICAQ2MRDKEEOX54R4NDGKDHR44DVCQXQPP5KAP64B7IY6A, seg_pos={0 0}, upload=197977]: Done uploading piece. err: <nil> Aug 16 15:45:00.369330872 (190h18m54.800411003s) [noise=true , address=104.36.82.9:4030 , piece_id=PX3CBZZUG7XQGH77N4LSSZG75GTC2FGRI5AUYQVX4YHLM2CPEOSQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: ecclient: upload failed (node:1MGCEjVE57ej5VuXgC6pYPFNUuAL851p2gqC53xxj6hEXvgYM5, address:104.36.82.9:4030): protocol: rpc: dial tcp 104.36.82.9:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:00.369350711 (190h18m54.800430842s) [noise=true , address=104.36.82.5:4868 , piece_id=OTY3Z37532MJFIZK4OHIS6WJP7HGJEEDVRMI7QQYQK55SSNEB6TA, seg_pos={0 0}, upload=197977]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: write tcp 188.166.225.170:59124->104.36.82.5:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:00.369374376 (190h18m54.800454507s) [noise=true , address=104.36.82.60:4364 , piece_id=JVMUDVXMWJPAPPLIA4LX2FUYBEMU5IUGG3SM3ACIUTVXR4OM25AA, seg_pos={0 0}, upload=197977]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:104.36.82.60:4364): protocol: rpc: dial tcp 104.36.82.60:4364: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:00.369389014 (190h18m54.800469145s) [noise=true , address=104.36.82.44:4902 , piece_id=JWTN6VMW6BOCJ755D46MJDHHOI5OX5R2GGRC6EVLMWZFSBRW4SLQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: write tcp 188.166.225.170:40204->104.36.82.44:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:00.369421306 (190h18m54.800501427s) [noise=true , address=104.36.82.4:4835 , piece_id=IPH2MRFOUNVFIA45XR4PR5P5VSMOR2WCLCFHOSBRES2M73SVE6CQ, seg_pos={0 0}, upload=197977]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:104.36.82.4:4835): protocol: write tcp 188.166.225.170:51556->104.36.82.4:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:00.369445162 (190h18m54.800525293s) [seg_pos={0 0}, upload=197977]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:45:00.712059938 (190h18m55.143140069s) [upload=197977]: Done uploading segments... Aug 16 15:45:00.712086379 (190h18m55.14316651s) [upload=197977]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.344328145s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 23.293802768s FAIL (2025-08-16 15:44:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.29381389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [X...X...X..................X.X..X.....X.....................] duration: 2.520888961s FAIL (2025-08-16 15:44:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.520917927s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.232934125s FAIL (2025-08-16 15:44:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ding piece... Aug 16 15:43:38.131209285 (190h17m32.562289416s) [noise=true , address=104.36.82.44:4875 , piece_id=F73DI4Z6U5Y6YNGYSR77ZLHFG2C3UMGKPU3QLKX6OYPIAXKFLDAQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.131356648 (190h17m32.562436779s) [noise=true , address=79.127.220.101:4076 , piece_id=6DBQHKJWKLIX6EPGEFKKRVDANE6MY6GOFLOG7I4WHBUIJSKMQNQQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.131546049 (190h17m32.56262618s) [noise=true , address=104.36.82.58:4274 , piece_id=U3EZC5WRX5FATZKRU4OLMRAGGT5GH5RFFQBRJGCT7N7GWYDEZMOQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.131866904 (190h17m32.562947035s) [noise=true , address=79.127.201.215:4344 , piece_id=Z6ZNMD62XFTZFY4LOY6POUMZJ56YCE5OZJNLNRGEVZFXS5ALOGDA, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.131884458 (190h17m32.562964589s) [noise=true , address=77.90.31.13:4656 , piece_id=CNJWV6UAQIUJMAM3NYVS5FIYVEQYJGI6FBZHJSQMEEPB52DW4FMQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132147701 (190h17m32.563227832s) [noise=true , address=104.36.82.23:4457 , piece_id=BCNP3V4QJRASUQWWEWKQATUVSJEKSWP7ISB5IHJBZN4NY6AY5INQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132216583 (190h17m32.563296714s) [noise=true , address=104.36.82.4:4806 , piece_id=VQMFHBQAY5JRZ5SZTNP4HLEGQHSKUGHDVHQTC4PVW5YSNRVGR6FA, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132426585 (190h17m32.563506716s) [noise=true , address=104.36.82.44:4871 , piece_id=6T4CP6DW33GENNL4G3IFQPDJ4GAV7LAAIBJJRZPJFKJNLGS4PWPA, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132541385 (190h17m32.563621516s) [noise=true , address=77.90.31.21:4887 , piece_id=PVGVVET2ZAYJF3YTHXF36AB6KUZFNGEE4CO6SA3FAS3B7QS4U7TA, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132685270 (190h17m32.563765401s) [noise=true , address=104.36.82.18:4395 , piece_id=QX72K3JDRHHISXKZDAJZHUS2J5VR6AKJL3H7QNJGL6T6YZDFKRIQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.132942802 (190h17m32.564022933s) [noise=true , address=104.36.82.60:4346 , piece_id=JV45WOSGC5BCXKTPMRE265AY7BVJORST3JDLGGCFJM47CU2BENIQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.133183964 (190h17m32.564264095s) [noise=true , address=45.91.116.12:4244 , piece_id=342ZRCI3S7S4CXNT2CF2Y52GBVW7JF2EESIX2MVNI2UBSRU4U2VQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.133420837 (190h17m32.564500968s) [noise=true , address=66.23.193.175:4564 , piece_id=EHXBEH4V4YW24EY5Z34ZLQIY7J7LGZSHSIN47BMZOY62OOI4LMJA, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:38.133640337 (190h17m32.564720468s) [noise=true , address=79.127.201.214:4319 , piece_id=ADNA5A3DPX5AX7NMRHKNQOS6ST3U3X4WFTEMOPE4ZFWAPM7GBXLQ, seg_pos={0 0}, upload=197970]: Uploading piece... Aug 16 15:43:39.278100708 (190h17m33.709180839s) [upload=197970]: Next returned nil segment Aug 16 15:43:39.278117068 (190h17m33.709197199s) [upload=197970]: Waiting for error group managing segments... Aug 16 15:43:39.993038212 (190h17m34.424118343s) [noise=true , address=79.127.220.101:4076 , piece_id=6DBQHKJWKLIX6EPGEFKKRVDANE6MY6GOFLOG7I4WHBUIJSKMQNQQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:40.124561447 (190h17m34.555641568s) [noise=true , address=77.90.31.15:4725 , piece_id=RV3MA5INNJY7NA5MTRWRXNOA5FSHSX62Z6KFT2H5DHNPSWTR5LCQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:40.190177796 (190h17m34.621257928s) [noise=true , address=77.90.31.21:4887 , piece_id=PVGVVET2ZAYJF3YTHXF36AB6KUZFNGEE4CO6SA3FAS3B7QS4U7TA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:41.162258848 (190h17m35.593338979s) [noise=true , address=79.127.201.214:4319 , piece_id=ADNA5A3DPX5AX7NMRHKNQOS6ST3U3X4WFTEMOPE4ZFWAPM7GBXLQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:41.636805951 (190h17m36.067886072s) [noise=true , address=45.91.116.12:4244 , piece_id=342ZRCI3S7S4CXNT2CF2Y52GBVW7JF2EESIX2MVNI2UBSRU4U2VQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:45.060959373 (190h17m39.492039494s) [noise=true , address=66.23.193.175:4564 , piece_id=EHXBEH4V4YW24EY5Z34ZLQIY7J7LGZSHSIN47BMZOY62OOI4LMJA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:43:45.666476424 (190h17m40.097556555s) [noise=true , address=79.127.201.215:4344 , piece_id=Z6ZNMD62XFTZFY4LOY6POUMZJ56YCE5OZJNLNRGEVZFXS5ALOGDA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: <nil> Aug 16 15:44:37.034808301 (190h18m31.465888422s) [noise=true , address=104.36.82.38:4720 , piece_id=ALES2Y4PKMX4YAFF5EUVUY5Q3XIPLNVY7GCN6EQ4WXLZ2NP72W3Q, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC, address:104.36.82.38:4720): protocol: rpc: dial tcp 104.36.82.38:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.034908383 (190h18m31.465988514s) [noise=true , address=104.36.82.23:4457 , piece_id=BCNP3V4QJRASUQWWEWKQATUVSJEKSWP7ISB5IHJBZN4NY6AY5INQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: rpc: dial tcp 104.36.82.23:4457: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.034952247 (190h18m31.466032378s) [noise=true , address=104.36.82.59:4305 , piece_id=CHNIT3VQMRQHVR6FCQ5KMNVAYWZIBJ4AQE3S7CJFZKAHWZN7UNFA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB, address:104.36.82.59:4305): protocol: rpc: dial tcp 104.36.82.59:4305: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.034982726 (190h18m31.466062847s) [noise=true , address=104.36.82.44:4875 , piece_id=F73DI4Z6U5Y6YNGYSR77ZLHFG2C3UMGKPU3QLKX6OYPIAXKFLDAQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:104.36.82.44:4875): protocol: rpc: dial tcp 104.36.82.44:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035012713 (190h18m31.466092834s) [noise=true , address=104.36.82.44:4871 , piece_id=6T4CP6DW33GENNL4G3IFQPDJ4GAV7LAAIBJJRZPJFKJNLGS4PWPA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035068479 (190h18m31.466148621s) [noise=true , address=104.36.82.60:4346 , piece_id=JV45WOSGC5BCXKTPMRE265AY7BVJORST3JDLGGCFJM47CU2BENIQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:104.36.82.60:4346): protocol: rpc: dial tcp 104.36.82.60:4346: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035138493 (190h18m31.466218624s) [noise=true , address=104.36.82.58:4274 , piece_id=U3EZC5WRX5FATZKRU4OLMRAGGT5GH5RFFQBRJGCT7N7GWYDEZMOQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:129cCwcw9NVVWnGekaWEGhSam1K3TZDcFVPdzZ32aQvP8kYVtZi, address:104.36.82.58:4274): protocol: rpc: dial tcp 104.36.82.58:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035268772 (190h18m31.466348903s) [noise=true , address=77.90.31.13:4656 , piece_id=CNJWV6UAQIUJMAM3NYVS5FIYVEQYJGI6FBZHJSQMEEPB52DW4FMQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: read tcp 188.166.225.170:47498->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035294882 (190h18m31.466375003s) [noise=true , address=104.36.82.4:4806 , piece_id=VQMFHBQAY5JRZ5SZTNP4HLEGQHSKUGHDVHQTC4PVW5YSNRVGR6FA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12qUyweKQhw15RjewCgKi1oLyCvN3VUB1VMpeGhj2Q2hszvJBsQ, address:104.36.82.4:4806): protocol: write tcp 188.166.225.170:60316->104.36.82.4:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035353805 (190h18m31.466433936s) [noise=true , address=104.36.82.18:4395 , piece_id=QX72K3JDRHHISXKZDAJZHUS2J5VR6AKJL3H7QNJGL6T6YZDFKRIQ, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:1GuYEUnNETTGNbrnMa2TBXMJaizRAUNrrC94przAGiSFyEi4B2, address:104.36.82.18:4395): protocol: write tcp 188.166.225.170:45050->104.36.82.18:4395: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:37.035418299 (190h18m31.46649843s) [noise=true , address=104.36.82.16:4303 , piece_id=ZZQ4OTDHNSB3GGKAMTFIZJ3N6T47PLEFSUHY7EZMG3THLCLH7BGA, seg_pos={0 0}, upload=197970]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:44:37.035457053 (190h18m31.466537184s) [seg_pos={0 0}, upload=197970]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:44:37.265571672 (190h18m31.696651793s) [upload=197970]: Done uploading segments... Aug 16 15:44:37.265598293 (190h18m31.696678424s) [upload=197970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.232949264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.819319737s FAIL (2025-08-16 15:44:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.81935235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.X...X...X..................X.X..X.....X....................] duration: 2.429246527s FAIL (2025-08-16 15:44:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.429283668s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...................................X......X..XXXXXXXXXX.XX.X] duration: 30.230808316s FAIL (2025-08-16 15:44:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:43:42.083932889 (190h17m36.51501302s) [upload=197974]: Starting upload Aug 16 15:43:42.084009887 (190h17m36.515090028s) [upload=197974]: Done starting upload Aug 16 15:43:42.084251990 (190h17m36.515332121s) [upload=197974]: Uploading segments... Aug 16 15:43:42.084317676 (190h17m36.515397817s) [upload=197974]: Got next segment. Inline: false Aug 16 15:43:42.084373973 (190h17m36.515454104s) [upload=197974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:42.084378202 (190h17m36.515458333s) [upload=197974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:42.322917179 (190h17m36.753997311s) [seg_pos={0 0}, upload=197974]: Begin upload segment... Aug 16 15:43:42.322963899 (190h17m36.75404403s) [seg_pos={0 0}, upload=197974]: Done begin upload segment. Aug 16 15:43:42.322984258 (190h17m36.754064389s) [noise=true , address=104.36.82.6:4876 , piece_id=KWA3DJ7DZWPUJCQHJNINLBMUP2J64TPEVH3NVEGRXYSFBMBF4OBQ, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.323082566 (190h17m36.754162687s) [noise=true , address=104.36.82.15:4259 , piece_id=PSQPQ26XQFUCHBPG2GF2CGVXDN45F67KCSFAK7DLXYY4RBS2YO7A, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.323454809 (190h17m36.75453494s) [noise=true , address=104.36.82.10:4077 , piece_id=V7ML44TMWHAIOCJX6VRCQXHH7Q63RYUABUEDUXCKXUD2XRJU645Q, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.323510345 (190h17m36.754590476s) [noise=true , address=104.36.82.9:4005 , piece_id=R22DXO7PJEL76HMOA37JI2G5LZJRP6SWPZMUNITMWFLZWF4SHIWA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.323750945 (190h17m36.754831076s) [noise=true , address=104.36.82.13:4185 , piece_id=GB6IEIC4Y4XCMD2625IFXDYKG3WA6EQ24AQK4DMGLSY4XOTX4GUA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.323773328 (190h17m36.754853459s) [noise=true , address=45.91.116.10:4082 , piece_id=RGHTZXTFFKZGPUXONTMNC5EFCRAA3G3OY7QWCJMJODVZ4QI2SI5A, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324032764 (190h17m36.755112895s) [noise=true , address=77.90.31.11:4555 , piece_id=AEBAD76X7BUPNYWE2AIOI3ASL46MQORVCC2DZKHAPPNQJPHTT2EQ, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324055578 (190h17m36.755135709s) [noise=true , address=45.91.116.5:4298 , piece_id=JRWEMBGS62BDAITEEMMZIVYUJZ63MF3J5XOXPXYYL362LASOS4MA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324269748 (190h17m36.755349879s) [noise=true , address=45.91.116.7:4009 , piece_id=AQCJ6IPHKYMBKMFZA6AW4FHX43QNJA3MEATWD63TBJSRLGVAIMZQ, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324323481 (190h17m36.755403612s) [noise=true , address=45.91.116.8:4047 , piece_id=YC25FJICXFVNYRBOQ5JDQP4WCZKCFM6F7POGZG4OCS4N4THPJ7XQ, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324554372 (190h17m36.755634493s) [noise=true , address=77.90.31.11:4554 , piece_id=UXZMUTT2WA6WKNGXZOMSZXCJC32VGIE2RPIIQ7F2FGLM47FQ633A, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324577707 (190h17m36.755657838s) [noise=true , address=45.91.116.14:4190 , piece_id=2LTK5QWHXLIGXGSR6OEYK6F7IQAKQ7OHSABCHCSYR4XKCPN2P5MA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.324813128 (190h17m36.755893259s) [noise=true , address=104.36.82.12:4150 , piece_id=3CHWSFRTE5HXYI5AX7EER4S5JVBZXPSJOU7MH2DC3QNASV6UN4UA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.325061543 (190h17m36.756141674s) [noise=true , address=77.90.31.13:4627 , piece_id=RPGNH2GSNGKQ4B5YDATNVGNGMAUXNX6OJDIPG37QDD2ZFDCNGWNA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.325256616 (190h17m36.756336747s) [noise=true , address=104.36.82.11:4113 , piece_id=DMDB56C6VZIZNTGE4WITZC5Y5U4VDS7VPKIABPWYPJKMPGP7VMBA, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:42.326343826 (190h17m36.757423957s) [noise=true , address=77.90.31.16:4733 , piece_id=XHGOWOLYXG2Q7P7DGC6FFWD6IVRXBJSCWHACUNYDSBTNFTKUS7FQ, seg_pos={0 0}, upload=197974]: Uploading piece... Aug 16 15:43:44.091321607 (190h17m38.522401738s) [upload=197974]: Next returned nil segment Aug 16 15:43:44.091338530 (190h17m38.522418651s) [upload=197974]: Waiting for error group managing segments... Aug 16 15:43:44.424473687 (190h17m38.855553848s) [noise=true , address=104.36.82.15:4259 , piece_id=PSQPQ26XQFUCHBPG2GF2CGVXDN45F67KCSFAK7DLXYY4RBS2YO7A, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.425072272 (190h17m38.856152453s) [noise=true , address=104.36.82.6:4876 , piece_id=KWA3DJ7DZWPUJCQHJNINLBMUP2J64TPEVH3NVEGRXYSFBMBF4OBQ, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.512970045 (190h17m38.944050176s) [noise=true , address=104.36.82.12:4150 , piece_id=3CHWSFRTE5HXYI5AX7EER4S5JVBZXPSJOU7MH2DC3QNASV6UN4UA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.535008964 (190h17m38.966089085s) [noise=true , address=104.36.82.10:4077 , piece_id=V7ML44TMWHAIOCJX6VRCQXHH7Q63RYUABUEDUXCKXUD2XRJU645Q, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.578783663 (190h17m39.009863824s) [noise=true , address=77.90.31.11:4555 , piece_id=AEBAD76X7BUPNYWE2AIOI3ASL46MQORVCC2DZKHAPPNQJPHTT2EQ, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.628098691 (190h17m39.059178822s) [noise=true , address=45.91.116.5:4298 , piece_id=JRWEMBGS62BDAITEEMMZIVYUJZ63MF3J5XOXPXYYL362LASOS4MA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.662822271 (190h17m39.093902392s) [noise=true , address=45.91.116.14:4190 , piece_id=2LTK5QWHXLIGXGSR6OEYK6F7IQAKQ7OHSABCHCSYR4XKCPN2P5MA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.666928669 (190h17m39.0980088s) [noise=true , address=77.90.31.11:4554 , piece_id=UXZMUTT2WA6WKNGXZOMSZXCJC32VGIE2RPIIQ7F2FGLM47FQ633A, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.819125100 (190h17m39.250205231s) [noise=true , address=45.91.116.7:4009 , piece_id=AQCJ6IPHKYMBKMFZA6AW4FHX43QNJA3MEATWD63TBJSRLGVAIMZQ, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.884972161 (190h17m39.316052292s) [noise=true , address=45.91.116.8:4047 , piece_id=YC25FJICXFVNYRBOQ5JDQP4WCZKCFM6F7POGZG4OCS4N4THPJ7XQ, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:44.891698722 (190h17m39.322778853s) [noise=true , address=45.91.116.10:4082 , piece_id=RGHTZXTFFKZGPUXONTMNC5EFCRAA3G3OY7QWCJMJODVZ4QI2SI5A, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:43:45.724894010 (190h17m40.155974141s) [noise=true , address=77.90.31.16:4733 , piece_id=XHGOWOLYXG2Q7P7DGC6FFWD6IVRXBJSCWHACUNYDSBTNFTKUS7FQ, seg_pos={0 0}, upload=197974]: Done uploading piece. err: <nil> Aug 16 15:44:11.436855387 (190h18m5.867935508s) [noise=true , address=77.90.31.13:4627 , piece_id=RPGNH2GSNGKQ4B5YDATNVGNGMAUXNX6OJDIPG37QDD2ZFDCNGWNA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: rpc: dial tcp 77.90.31.13:4627: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:11.436903139 (190h18m5.86798327s) [noise=true , address=104.36.82.13:4185 , piece_id=GB6IEIC4Y4XCMD2625IFXDYKG3WA6EQ24AQK4DMGLSY4XOTX4GUA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:11.436931693 (190h18m5.868011824s) [noise=true , address=104.36.82.11:4113 , piece_id=DMDB56C6VZIZNTGE4WITZC5Y5U4VDS7VPKIABPWYPJKMPGP7VMBA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:11.436965227 (190h18m5.868045348s) [noise=true , address=104.36.82.9:4005 , piece_id=R22DXO7PJEL76HMOA37JI2G5LZJRP6SWPZMUNITMWFLZWF4SHIWA, seg_pos={0 0}, upload=197974]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:36262->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:11.436979374 (190h18m5.868059495s) [seg_pos={0 0}, upload=197974]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:44:11.665940877 (190h18m6.097021008s) [upload=197974]: Done uploading segments... Aug 16 15:44:11.665973840 (190h18m6.097053971s) [upload=197974]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230828806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000882714s FAIL (2025-08-16 15:44:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000906039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [........X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXX] duration: 1m0.231212693s FAIL (2025-08-16 15:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Aug 16 15:42:47.269596486 (190h16m41.700676617s) [upload=197959]: Starting upload Aug 16 15:42:47.269661320 (190h16m41.700741451s) [upload=197959]: Done starting upload Aug 16 15:42:47.269774927 (190h16m41.700855058s) [upload=197959]: Uploading segments... Aug 16 15:42:47.269920184 (190h16m41.701000316s) [upload=197959]: Got next segment. Inline: false Aug 16 15:42:47.269951666 (190h16m41.701031797s) [upload=197959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:42:47.269958508 (190h16m41.701038639s) [upload=197959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:42:47.777583622 (190h16m42.208663743s) [seg_pos={0 0}, upload=197959]: Begin upload segment... Aug 16 15:42:47.777646162 (190h16m42.208726293s) [seg_pos={0 0}, upload=197959]: Done begin upload segment. Aug 16 15:42:47.777665629 (190h16m42.20874576s) [noise=true , address=77.90.31.19:4846 , piece_id=I2RETN3WPIHKFJJI565ZZXVCUFHLHL6FZGBJLTVTEBSHW5HJD7KQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.777721185 (190h16m42.208801316s) [noise=true , address=77.90.31.11:4585 , piece_id=6ZRMSH6XQKBNKQLEZERH7IFSE3HPEBZBKFLW3SSHP4TCHADHE7CQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778030347 (190h16m42.209110478s) [noise=true , address=79.127.205.246:4259 , piece_id=ACD6R3SDH6HXROLHZBKJRIN6IXIJ32MUU6ANQNOJIEF6N4STEC2A, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778220171 (190h16m42.209300302s) [noise=true , address=79.127.201.215:4345 , piece_id=4ZV62SHBXG3YI25Z44QS6NTCRCJ5H537WC4NKLF3IZMCG4EGQ4XA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778312968 (190h16m42.209393099s) [noise=true , address=104.36.82.5:4872 , piece_id=OEWO633ZLW6EIIXDCIB3JAE3N2PLN5LKZBVNR7WEJQLZIGRGKPIQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778552687 (190h16m42.209632818s) [noise=true , address=79.127.201.214:4319 , piece_id=2IRUOA52PAM6XYNEHIMJIA4HDATTJ2QMWLCTNUVWTBRINXLNHVCA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778668438 (190h16m42.209748639s) [noise=true , address=104.36.82.38:4709 , piece_id=PIY662CN3SJWUS5ETCANCD5JFGIOVB2XOMWS7OP56OMUIDS7QZ7Q, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.778871256 (190h16m42.209951387s) [noise=true , address=104.36.82.35:4591 , piece_id=XBJW6G6DOSJHVWKJAOJWUCTNYNVNXX33TZTPUZ3RHNSAJ7Y7D75A, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779036311 (190h16m42.210116442s) [noise=true , address=104.36.82.36:4636 , piece_id=HAUYWU74ME6LOTVE6ZZ6TN7T4EIDLF2DEKGBHEQAWSGDL7LVDU7Q, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779153356 (190h16m42.210233487s) [noise=true , address=104.36.82.23:4446 , piece_id=MQ5NKMMV7QBEXN4ZSWKWTP6OZW55ELDX6WB2FFVWURMNDPWB26PQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779260491 (190h16m42.210340622s) [noise=true , address=77.90.31.10:4541 , piece_id=2ZQI77PZPRSLNHYBO7BP3CQX52G3RMPBUE4SICRG77CZDJEDBFXQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779412802 (190h16m42.210492933s) [noise=true , address=79.127.220.97:4392 , piece_id=2BHLIJZST43I6QUUYOTOZSNQ7AYSMVZEKBM4HJ2GZFTKPH6WQZPQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779475201 (190h16m42.210555322s) [noise=true , address=104.36.82.44:4896 , piece_id=2J5ABIGPULGQDDRNNEG6LXOK4DXKVGZ5YFDIIOZU5KQKBCJDD2RA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779691355 (190h16m42.210771476s) [noise=true , address=45.91.116.14:4198 , piece_id=ASS3D7WYKQJPUEIXTASTN2COLMFNRWLOESA7YXNV6CNTSJUZWNRA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779700082 (190h16m42.210780213s) [noise=true , address=77.90.31.21:4890 , piece_id=WNGC7MKZMH3QZZFGXRRKWP5AXLCDASFIQ2AAZIACEYTIWPUUW5GQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779943558 (190h16m42.211023689s) [noise=true , address=77.90.31.18:4830 , piece_id=WJJOFFVXWIYFAOBHQ5B4N3WQ6XHTF67PL5VYN74RAYM5ZJK65GZA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.779998423 (190h16m42.211078554s) [noise=true , address=104.36.82.60:4367 , piece_id=SR6QLW7DY5JUM2TWM5AYNPHDGA3WUQ724YYYDW7MNCTZIIA576NA, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:47.780275624 (190h16m42.211355755s) [noise=true , address=77.90.31.14:4665 , piece_id=FMVNSCV42XKNKZBSNPZQCGJQVJEMT3RM4J2U32AQ2PCZ53GQ7DXQ, seg_pos={0 0}, upload=197959]: Uploading piece... Aug 16 15:42:48.541056324 (190h16m42.972136455s) [noise=true , address=79.127.205.246:4259 , piece_id=ACD6R3SDH6HXROLHZBKJRIN6IXIJ32MUU6ANQNOJIEF6N4STEC2A, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1KP7PTvd2bAn6khSD6VUWeP4SCmJikD9iY3NKt5axV7VeaUeDa, address:79.127.205.246:4259): protocol: storage node overloaded, request limit: 2000 Aug 16 15:42:49.287183076 (190h16m43.718263207s) [upload=197959]: Next returned nil segment Aug 16 15:42:49.287200238 (190h16m43.718280359s) [upload=197959]: Waiting for error group managing segments... Aug 16 15:42:49.860974573 (190h16m44.292054694s) [noise=true , address=79.127.220.97:4392 , piece_id=2BHLIJZST43I6QUUYOTOZSNQ7AYSMVZEKBM4HJ2GZFTKPH6WQZPQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:42:51.166184951 (190h16m45.597265082s) [noise=true , address=79.127.201.214:4319 , piece_id=2IRUOA52PAM6XYNEHIMJIA4HDATTJ2QMWLCTNUVWTBRINXLNHVCA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:42:51.755704170 (190h16m46.186784301s) [noise=true , address=77.90.31.14:4665 , piece_id=FMVNSCV42XKNKZBSNPZQCGJQVJEMT3RM4J2U32AQ2PCZ53GQ7DXQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:42:52.055018876 (190h16m46.486099007s) [noise=true , address=77.90.31.21:4890 , piece_id=WNGC7MKZMH3QZZFGXRRKWP5AXLCDASFIQ2AAZIACEYTIWPUUW5GQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:42:52.191229661 (190h16m46.622309792s) [noise=true , address=77.90.31.11:4585 , piece_id=6ZRMSH6XQKBNKQLEZERH7IFSE3HPEBZBKFLW3SSHP4TCHADHE7CQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:42:54.608978571 (190h16m49.040058712s) [noise=true , address=79.127.201.215:4345 , piece_id=4ZV62SHBXG3YI25Z44QS6NTCRCJ5H537WC4NKLF3IZMCG4EGQ4XA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:43:06.504929031 (190h17m0.936009162s) [noise=true , address=77.90.31.19:4846 , piece_id=I2RETN3WPIHKFJJI565ZZXVCUFHLHL6FZGBJLTVTEBSHW5HJD7KQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:43:36.816586809 (190h17m31.247666941s) [noise=true , address=45.91.116.14:4198 , piece_id=ASS3D7WYKQJPUEIXTASTN2COLMFNRWLOESA7YXNV6CNTSJUZWNRA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: <nil> Aug 16 15:43:46.611803983 (190h17m41.042884114s) [noise=true , address=104.36.82.44:4896 , piece_id=2J5ABIGPULGQDDRNNEG6LXOK4DXKVGZ5YFDIIOZU5KQKBCJDD2RA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 188.166.225.170:33478->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611824532 (190h17m41.042904663s) [noise=true , address=77.90.31.10:4541 , piece_id=2ZQI77PZPRSLNHYBO7BP3CQX52G3RMPBUE4SICRG77CZDJEDBFXQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: rpc: dial tcp 77.90.31.10:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611860350 (190h17m41.042940472s) [noise=true , address=104.36.82.60:4367 , piece_id=SR6QLW7DY5JUM2TWM5AYNPHDGA3WUQ724YYYDW7MNCTZIIA576NA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK, address:104.36.82.60:4367): protocol: rpc: dial tcp 104.36.82.60:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611874518 (190h17m41.042954649s) [noise=true , address=104.36.82.36:4636 , piece_id=HAUYWU74ME6LOTVE6ZZ6TN7T4EIDLF2DEKGBHEQAWSGDL7LVDU7Q, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1Lx4f6jtUdcZd9mVy5DgWqZ2vd4SrpSviaT7WUedxs7PLyzfHZ, address:104.36.82.36:4636): protocol: rpc: dial tcp 104.36.82.36:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611902111 (190h17m41.042982242s) [noise=true , address=104.36.82.5:4872 , piece_id=OEWO633ZLW6EIIXDCIB3JAE3N2PLN5LKZBVNR7WEJQLZIGRGKPIQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: rpc: dial tcp 104.36.82.5:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611912621 (190h17m41.042992742s) [noise=true , address=104.36.82.23:4446 , piece_id=MQ5NKMMV7QBEXN4ZSWKWTP6OZW55ELDX6WB2FFVWURMNDPWB26PQ, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: rpc: dial tcp 104.36.82.23:4446: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611942337 (190h17m41.043022468s) [noise=true , address=104.36.82.35:4591 , piece_id=XBJW6G6DOSJHVWKJAOJWUCTNYNVNXX33TZTPUZ3RHNSAJ7Y7D75A, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55, address:104.36.82.35:4591): protocol: write tcp 188.166.225.170:59022->104.36.82.35:4591: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.611994317 (190h17m41.043074438s) [noise=true , address=104.36.82.38:4709 , piece_id=PIY662CN3SJWUS5ETCANCD5JFGIOVB2XOMWS7OP56OMUIDS7QZ7Q, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: write tcp 188.166.225.170:53182->104.36.82.38:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:46.612019054 (190h17m41.043099185s) [noise=true , address=77.90.31.18:4830 , piece_id=WJJOFFVXWIYFAOBHQ5B4N3WQ6XHTF67PL5VYN74RAYM5ZJK65GZA, seg_pos={0 0}, upload=197959]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: read tcp 188.166.225.170:51888->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:43:46.612057779 (190h17m41.04313791s) [seg_pos={0 0}, upload=197959]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:43:46.840592457 (190h17m41.271672678s) [upload=197959]: Done uploading segments... Aug 16 15:43:46.840625280 (190h17m41.271705491s) [upload=197959]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.231226s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.22775041s FAIL (2025-08-16 15:43:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== :42:37.671510659 (190h16m32.10259079s) [noise=true , address=104.36.82.39:4733 , piece_id=JJJKBI3HLUF44F74TGNTOIJHSC4X4TNLPIXJRTP54P4LXZ57KYMA, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.672036825 (190h16m32.103116956s) [noise=true , address=104.36.82.44:4873 , piece_id=CRWQPGELF5CADEPS3UDXGU5MTLYMQMIQAOXPWHMPSBGTA5ASBLXQ, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.672674245 (190h16m32.103754376s) [noise=true , address=77.90.31.21:4908 , piece_id=AJDTWG4BKXK4X3MW3ZYCI2PXIXLVET2WGA2WL6W4BS2ONVUNSIEQ, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.673597421 (190h16m32.104677552s) [noise=true , address=104.36.82.58:4289 , piece_id=FUIUUG5IOC5ES5HKELLNQFMGRSK67LAI3UBMPFEDBI2PW4JL5IIA, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.673960426 (190h16m32.105040557s) [noise=true , address=104.36.82.3:4799 , piece_id=JK6KLLZA2TQM45DAYL5JRETL4P4BRYYRLDGHDIUSNEG3WJLXTOSA, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.674208009 (190h16m32.10528814s) [noise=true , address=156.146.43.225:4457 , piece_id=7EX4XUYBDNG3NFCKQ6RXPAW75TNXRFWZJ5TFHFBZGC5JIOO4WS5A, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.674434623 (190h16m32.105514754s) [noise=true , address=104.36.82.34:4559 , piece_id=AX7HSTJOPORPV6YSWBRDDDWFOVECTRYW7JBD5AITSDDVFYEUH6GA, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.674662770 (190h16m32.105742901s) [noise=true , address=104.36.82.38:4715 , piece_id=GE5VPA6RAVW4I3E7L6R4GZANQOUHMHUSNZCWXF74DT6VKZLIWX4A, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.674854517 (190h16m32.105934648s) [noise=true , address=79.127.201.214:4319 , piece_id=DY63EAIYZMWS45W7AIWCH3G3MLNKOTRBJYAE4TK4PJGMDWO5QVRA, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.675071242 (190h16m32.106151373s) [noise=true , address=104.36.82.59:4322 , piece_id=NXD3NOAXUJZPE5YJJW6EA22NMLPYHTN3MVPMGMDNTNGUN7DH5X7Q, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.675261766 (190h16m32.106341887s) [noise=true , address=104.36.82.60:4365 , piece_id=7DJE7E7CVARIT2KJO4QJMABJVT4VLGRDDWFINZFMOMHHVS73DZBQ, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:37.675450026 (190h16m32.106530157s) [noise=true , address=77.90.31.15:4730 , piece_id=GU5LURAKTVGRJDHKNYYUTKM7B5Z6GR42LXSIGAS7UPS6JY7OGMQQ, seg_pos={0 0}, upload=197957]: Uploading piece... Aug 16 15:42:38.898526219 (190h16m33.32960635s) [upload=197957]: Next returned nil segment Aug 16 15:42:38.898543091 (190h16m33.329623222s) [upload=197957]: Waiting for error group managing segments... Aug 16 15:42:39.821385288 (190h16m34.252465449s) [noise=true , address=66.23.193.175:4561 , piece_id=SRHSBNJAO63CPHV7PEVZR4ECSJEVEXI7X3VHKTD5BRIW4BD4SE6A, seg_pos={0 0}, upload=197957]: Done uploading piece. err: <nil> Aug 16 15:42:41.398348976 (190h16m35.829429107s) [noise=true , address=79.127.201.214:4319 , piece_id=DY63EAIYZMWS45W7AIWCH3G3MLNKOTRBJYAE4TK4PJGMDWO5QVRA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: <nil> Aug 16 15:42:42.294406814 (190h16m36.725486946s) [noise=true , address=156.146.43.225:4457 , piece_id=7EX4XUYBDNG3NFCKQ6RXPAW75TNXRFWZJ5TFHFBZGC5JIOO4WS5A, seg_pos={0 0}, upload=197957]: Done uploading piece. err: <nil> Aug 16 15:42:50.884738554 (190h16m45.315818685s) [noise=true , address=77.90.31.15:4730 , piece_id=GU5LURAKTVGRJDHKNYYUTKM7B5Z6GR42LXSIGAS7UPS6JY7OGMQQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: <nil> Aug 16 15:42:51.299763248 (190h16m45.73084338s) [noise=true , address=77.90.31.21:4908 , piece_id=AJDTWG4BKXK4X3MW3ZYCI2PXIXLVET2WGA2WL6W4BS2ONVUNSIEQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: <nil> Aug 16 15:43:36.571676250 (190h17m31.002756381s) [noise=true , address=77.90.31.13:4653 , piece_id=5Y44FVNMAKE2ATW4DNYZDVGHIHITOCTNKXNGP7NSL7ZW4SLKJ3PQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:43:36.572034365 (190h17m31.003114486s) [noise=true , address=104.36.82.18:4400 , piece_id=L3A2PGFARIMCHKSDGPBYVWBXBCDCCXNCNF2KS24XXVCSD4C6BLYA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12UxA6DMP2dr2T8o352iih2D6UeEeGQBMm1SgPbHYHuWaSFdyYv, address:104.36.82.18:4400): protocol: rpc: dial tcp 104.36.82.18:4400: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572072638 (190h17m31.003152769s) [noise=true , address=104.36.82.59:4322 , piece_id=NXD3NOAXUJZPE5YJJW6EA22NMLPYHTN3MVPMGMDNTNGUN7DH5X7Q, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12bELQDDH69yamGywE32K8YhpPbMRVorkpm85ZFuAtb2CD58ZPH, address:104.36.82.59:4322): protocol: rpc: dial tcp 104.36.82.59:4322: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572100872 (190h17m31.003181003s) [noise=true , address=104.36.82.60:4365 , piece_id=7DJE7E7CVARIT2KJO4QJMABJVT4VLGRDDWFINZFMOMHHVS73DZBQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572116993 (190h17m31.003197125s) [noise=true , address=104.36.82.44:4873 , piece_id=CRWQPGELF5CADEPS3UDXGU5MTLYMQMIQAOXPWHMPSBGTA5ASBLXQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: rpc: dial tcp 104.36.82.44:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572133454 (190h17m31.003213575s) [noise=true , address=77.90.31.18:4835 , piece_id=Q5NS3DQYIJR6BY44E6E4DGGPMTMIAHD3RR3YVYLYR2OUKDPDOP3A, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572149666 (190h17m31.003229797s) [noise=true , address=104.36.82.58:4289 , piece_id=FUIUUG5IOC5ES5HKELLNQFMGRSK67LAI3UBMPFEDBI2PW4JL5IIA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:1v4QJpSy7hk4pShMu3FswnZAf3rzyCNUx5sUZSPSvxpecY8T9X, address:104.36.82.58:4289): protocol: write tcp 188.166.225.170:38334->104.36.82.58:4289: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572165947 (190h17m31.003246068s) [noise=true , address=104.36.82.23:4453 , piece_id=3GT7S7Z2TA6WKCDSTH66MEBKYIBQX6L52OQADUEN2EG2JLRFGCAQ, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572184823 (190h17m31.003264954s) [noise=true , address=104.36.82.3:4799 , piece_id=JK6KLLZA2TQM45DAYL5JRETL4P4BRYYRLDGHDIUSNEG3WJLXTOSA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: rpc: dial tcp 104.36.82.3:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572199251 (190h17m31.003279382s) [noise=true , address=104.36.82.39:4733 , piece_id=JJJKBI3HLUF44F74TGNTOIJHSC4X4TNLPIXJRTP54P4LXZ57KYMA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:12FW6aFPa6hXMda3wrhHCWj5tXtBskLta5objDs1kP15Szf85H, address:104.36.82.39:4733): protocol: write tcp 188.166.225.170:55522->104.36.82.39:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572217345 (190h17m31.003297476s) [noise=true , address=104.36.82.38:4715 , piece_id=GE5VPA6RAVW4I3E7L6R4GZANQOUHMHUSNZCWXF74DT6VKZLIWX4A, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: rpc: dial tcp 104.36.82.38:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572224209 (190h17m31.00330434s) [noise=true , address=104.36.82.34:4559 , piece_id=AX7HSTJOPORPV6YSWBRDDDWFOVECTRYW7JBD5AITSDDVFYEUH6GA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:1XXGQfMeXYCUuSh5TUe8pTwwd19wn68r9buS11N8aJdx6vZysf, address:104.36.82.34:4559): protocol: rpc: dial tcp 104.36.82.34:4559: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572235199 (190h17m31.00331533s) [noise=true , address=77.90.31.10:4541 , piece_id=SARLSKXVVVDSQXTRNJR4VJE7JYOFVRXAA7EL4YXFGUIIOYADOFMA, seg_pos={0 0}, upload=197957]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 188.166.225.170:54182->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:36.572254557 (190h17m31.003334688s) [seg_pos={0 0}, upload=197957]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Aug 16 15:43:36.797725019 (190h17m31.22880515s) [upload=197957]: Done uploading segments... Aug 16 15:43:36.797757401 (190h17m31.228837532s) [upload=197957]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.227761981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [..X...X...X..................X.X..X.....X...................] duration: 2.67627568s FAIL (2025-08-16 15:43:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.676298805s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................................X......X..XXXXXXXXXX.XX.] duration: 27.955298186s FAIL (2025-08-16 15:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:41:45.100124180 (190h15m39.531204311s) [upload=197941]: Starting upload Aug 16 15:41:45.100231585 (190h15m39.531311716s) [upload=197941]: Done starting upload Aug 16 15:41:45.100352096 (190h15m39.531432227s) [upload=197941]: Uploading segments... Aug 16 15:41:45.100588929 (190h15m39.53166906s) [upload=197941]: Got next segment. Inline: false Aug 16 15:41:45.100630899 (190h15m39.53171102s) [upload=197941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:45.100636049 (190h15m39.53171618s) [upload=197941]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:45.354341927 (190h15m39.785422058s) [seg_pos={0 0}, upload=197941]: Begin upload segment... Aug 16 15:41:45.354398555 (190h15m39.785478676s) [seg_pos={0 0}, upload=197941]: Done begin upload segment. Aug 16 15:41:45.354426589 (190h15m39.78550672s) [noise=true , address=104.36.82.11:4114 , piece_id=IIE6IDPCC5AAQ2IX5L3LJSACNUWW6SA7D6VADS7ZU5P4RARGBWEQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.354451016 (190h15m39.785531147s) [noise=true , address=45.91.116.4:4937 , piece_id=QNNFWJ2HXISOFXN6FGMW2NE5KLDFHWKU7I7LBKBE6CBROAWB75ZQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.354822868 (190h15m39.785902999s) [noise=true , address=104.36.82.11:4115 , piece_id=IGLJ4UP4UDPSGWLKB43BMAEUYUSSB3TOY5TBGV2TWIOL42DZC6MQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355005598 (190h15m39.786085719s) [noise=true , address=104.36.82.15:4259 , piece_id=DEOBBLNEZNPP3HA7L4FZPB4ONPPBWWYGBZLCRAJYJD3GKUXQEJVQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355132720 (190h15m39.786212851s) [noise=true , address=104.36.82.7:4909 , piece_id=WPSE2E6WSD4ZIUKP7PYZPOJQAN2CJYAKJLUZ3KQX342TV6MGH36A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355301553 (190h15m39.786381684s) [noise=true , address=77.90.31.13:4628 , piece_id=HRGBGIN6GWHP5333NQ5XDJQONCJVFJW2CIMVEVBT3VTOBZRPXW4A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355432855 (190h15m39.786512976s) [noise=true , address=104.36.82.8:4969 , piece_id=6FNXQ3W5WMBFVDKOC3AMLUB7NGUK45UCIXQNZGR4US3JONH7H3HA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355747998 (190h15m39.786828129s) [noise=true , address=77.90.31.10:4518 , piece_id=QHFSQLVPFURVJ6PLYPNPHYEMN7EYRBTCKE5OQ6IQWULRAAVFV5DQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356065455 (190h15m39.787145586s) [noise=true , address=45.91.116.8:4047 , piece_id=SDZQ2IV3CB4W2KDJ3PXZWSV77IDB7IHLEZTFHHRU6OSTX32Q7JTA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356324761 (190h15m39.787404892s) [noise=true , address=104.36.82.7:4912 , piece_id=TKXZVAHQJM5RVW6OD7VAL2RZIX6Z5XS5TS2XM76CCSP3I6QL4NLQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356562967 (190h15m39.787643098s) [noise=true , address=104.36.82.12:4151 , piece_id=7H5WSMJPW2Q676WNHCITVIRF3K6T745ZSCRW2QTEAQ4SQU2KDQRA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356787115 (190h15m39.787867247s) [noise=true , address=104.36.82.6:4874 , piece_id=IIVXF7WUEDCRYCTRBQTKM4FJROW4U6WKGDVA7BAGR2ILRF2Q4ABQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357044529 (190h15m39.78812466s) [noise=true , address=104.36.82.10:4079 , piece_id=HIAOJGHNSH6L434KH4MDOTHT52XOOKPQE2ZW3KNB3ENW7KPGFGNQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357263728 (190h15m39.788343859s) [noise=true , address=45.91.116.12:4227 , piece_id=RIH4WNWQJYQLXW4COZSCQJG7EDVBTZPFTIJ3EKHYUT4T3M66IVAA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357499458 (190h15m39.78857959s) [noise=true , address=104.36.82.13:4187 , piece_id=YS2JY5V7FRLNURFGFPWAOOR6ZFIVGVJ6PI7C2DIRSAHLEI6F545A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357729038 (190h15m39.788809159s) [noise=true , address=77.90.31.16:4735 , piece_id=K2IGNDSGY7PXM4SMMDB4OPFRTIFTIW4IJJSDGPCGDIMLDKALHFZA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:47.119119724 (190h15m41.550199855s) [upload=197941]: Next returned nil segment Aug 16 15:41:47.119137288 (190h15m41.550217419s) [upload=197941]: Waiting for error group managing segments... Aug 16 15:41:47.402462310 (190h15m41.833542431s) [noise=true , address=104.36.82.7:4909 , piece_id=WPSE2E6WSD4ZIUKP7PYZPOJQAN2CJYAKJLUZ3KQX342TV6MGH36A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.419585042 (190h15m41.850665163s) [noise=true , address=104.36.82.10:4079 , piece_id=HIAOJGHNSH6L434KH4MDOTHT52XOOKPQE2ZW3KNB3ENW7KPGFGNQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.489994573 (190h15m41.921074714s) [noise=true , address=104.36.82.12:4151 , piece_id=7H5WSMJPW2Q676WNHCITVIRF3K6T745ZSCRW2QTEAQ4SQU2KDQRA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.503742047 (190h15m41.934822178s) [noise=true , address=104.36.82.15:4259 , piece_id=DEOBBLNEZNPP3HA7L4FZPB4ONPPBWWYGBZLCRAJYJD3GKUXQEJVQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.521485137 (190h15m41.952565268s) [noise=true , address=104.36.82.6:4874 , piece_id=IIVXF7WUEDCRYCTRBQTKM4FJROW4U6WKGDVA7BAGR2ILRF2Q4ABQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:49.572924071 (190h15m44.004004202s) [noise=true , address=104.36.82.7:4912 , piece_id=TKXZVAHQJM5RVW6OD7VAL2RZIX6Z5XS5TS2XM76CCSP3I6QL4NLQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:54.705143870 (190h15m49.136224002s) [noise=true , address=45.91.116.4:4937 , piece_id=QNNFWJ2HXISOFXN6FGMW2NE5KLDFHWKU7I7LBKBE6CBROAWB75ZQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:56.697020750 (190h15m51.128100871s) [noise=true , address=45.91.116.12:4227 , piece_id=RIH4WNWQJYQLXW4COZSCQJG7EDVBTZPFTIJ3EKHYUT4T3M66IVAA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:57.897503371 (190h15m52.328583503s) [noise=true , address=77.90.31.16:4735 , piece_id=K2IGNDSGY7PXM4SMMDB4OPFRTIFTIW4IJJSDGPCGDIMLDKALHFZA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:42:02.088178699 (190h15m56.51925883s) [noise=true , address=45.91.116.8:4047 , piece_id=SDZQ2IV3CB4W2KDJ3PXZWSV77IDB7IHLEZTFHHRU6OSTX32Q7JTA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:42:14.433410600 (190h16m8.864490721s) [noise=true , address=77.90.31.13:4628 , piece_id=HRGBGIN6GWHP5333NQ5XDJQONCJVFJW2CIMVEVBT3VTOBZRPXW4A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:42:14.433608378 (190h16m8.864688509s) [noise=true , address=77.90.31.10:4518 , piece_id=QHFSQLVPFURVJ6PLYPNPHYEMN7EYRBTCKE5OQ6IQWULRAAVFV5DQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433646671 (190h16m8.864726802s) [noise=true , address=104.36.82.11:4114 , piece_id=IIE6IDPCC5AAQ2IX5L3LJSACNUWW6SA7D6VADS7ZU5P4RARGBWEQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:44480->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433688491 (190h16m8.864768612s) [noise=true , address=104.36.82.11:4115 , piece_id=IGLJ4UP4UDPSGWLKB43BMAEUYUSSB3TOY5TBGV2TWIOL42DZC6MQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433716796 (190h16m8.864796927s) [noise=true , address=104.36.82.8:4969 , piece_id=6FNXQ3W5WMBFVDKOC3AMLUB7NGUK45UCIXQNZGR4US3JONH7H3HA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433728437 (190h16m8.864808568s) [noise=true , address=104.36.82.13:4187 , piece_id=YS2JY5V7FRLNURFGFPWAOOR6ZFIVGVJ6PI7C2DIRSAHLEI6F545A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433747454 (190h16m8.864827585s) [seg_pos={0 0}, upload=197941]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:42:14.659856879 (190h16m9.09093702s) [upload=197941]: Done uploading segments... Aug 16 15:42:14.659898739 (190h16m9.090978901s) [upload=197941]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 27.955326099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.........X......X...X.X......X..X......X..X...XXXXXXXXXXXXXX] duration: 1m0.377759901s FAIL (2025-08-16 15:42:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== Aug 16 15:41:46.666106052 (190h15m41.097186183s) [upload=197943]: Starting upload Aug 16 15:41:46.666159194 (190h15m41.097239315s) [upload=197943]: Done starting upload Aug 16 15:41:46.666272540 (190h15m41.097352672s) [upload=197943]: Uploading segments... Aug 16 15:41:46.666367453 (190h15m41.097447584s) [upload=197943]: Got next segment. Inline: false Aug 16 15:41:46.666416886 (190h15m41.097497017s) [upload=197943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:46.666421516 (190h15m41.097501657s) [upload=197943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:47.303726620 (190h15m41.734806751s) [seg_pos={0 0}, upload=197943]: Begin upload segment... Aug 16 15:41:47.303767127 (190h15m41.734847258s) [seg_pos={0 0}, upload=197943]: Done begin upload segment. Aug 16 15:41:47.303809939 (190h15m41.73489007s) [noise=true , address=45.91.116.12:4243 , piece_id=JHDTH5WCIV7ORESAAGNJJPESKYZRJ3WSPGVQSPXRMWIR6WMVIDMA, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.304239350 (190h15m41.735319481s) [noise=true , address=77.90.31.13:4649 , piece_id=LMCC6M7B2KNNMEVJGLX2PZUJ72QEI5J6GHY2L5VLG5IH5HYK4OLA, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.304605171 (190h15m41.735685292s) [noise=true , address=104.36.82.59:4329 , piece_id=CCO5T2GLDXOPETVASOINP5UAE2TMX3DRQ7HWY6322RG2WVN4YFPA, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.304661929 (190h15m41.73574206s) [noise=true , address=104.36.82.2:4763 , piece_id=YHZ3OZ3DWPHIWECFQU3RTBXQS3J2NAWARUA5ZZVIMJWYNUV6OFTQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.305105358 (190h15m41.736185489s) [noise=true , address=104.36.82.19:4416 , piece_id=6VBLTHMANUA7ONGXKPI6OSTPKKK2XUZOEVT55TTT3L3W2NK323MQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.305176944 (190h15m41.736257065s) [noise=true , address=104.36.82.34:4554 , piece_id=IUPTOB2YWGCIXTO4PSPNV3ERVOI2O3XYFJHBQJNZKK2SWW2OBXKQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.305716336 (190h15m41.736796457s) [noise=true , address=77.90.31.21:4902 , piece_id=CXXYON6ARXJQ7326EGNYV6IWUIEJB2Z6BMXC5P7LZCETM6O7EEMQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.305788525 (190h15m41.736868656s) [noise=true , address=79.127.201.215:4345 , piece_id=D3FCD6NLHJD2YW75L546PV5ARE3KTEMOD2RTU4PKCLFYZANFB4AQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.306057089 (190h15m41.73713722s) [noise=true , address=104.36.82.5:4844 , piece_id=VQ4ZM5M3X7XF6FYYGA3TUJTOSQKYMF6DZGWUZFRCOAK3YR22JZ3Q, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.306271028 (190h15m41.737351159s) [noise=true , address=104.36.82.3:4796 , piece_id=W2H4HO7T3RRHYAC4LLUR56MLRZPSWI7FQBMR5JTBTBCJBPTUWB4A, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.306465439 (190h15m41.737545571s) [noise=true , address=45.91.116.8:4072 , piece_id=TA4CVWYUYDINPJ7CAMFOO2BIOX5QRDB3ENDJ6RKOMAXQIZXMPVAQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.306691112 (190h15m41.737771243s) [noise=true , address=104.36.82.35:4603 , piece_id=PJC3J4F74FKJRQZGVOYLZJWROETWZRVK255K3ISR4V2WQ2ELHTQQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.306899600 (190h15m41.737979731s) [noise=true , address=104.36.82.60:4364 , piece_id=3P42GNO3VEN7MFTPUVURGJ7AB7EBJH6SMVK7JDH3QKYU2JLVJIZQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.307716704 (190h15m41.738796835s) [noise=true , address=104.36.82.44:4880 , piece_id=JJVEVORNQA4FFDA22LSSKR23CLWKK6WUSE3BJNQIV7QF6W7W323Q, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.308000567 (190h15m41.739080698s) [noise=true , address=104.36.82.40:4972 , piece_id=KWW3B4R4LI37B3EZNV446LORQRBA56SD7TPLROEY2O7NJICNTXGA, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.308245676 (190h15m41.739325807s) [noise=true , address=104.36.82.23:4447 , piece_id=UGUT723Q2NQPIOSIKKRCEUURJQNZD2YEJISIP427ITELPHQFXJJQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.308548726 (190h15m41.739628857s) [noise=true , address=66.23.193.175:4559 , piece_id=BMQZYYHQS2BEO4SU6UIBU5EPD4MUPDRQMKPS26AS7BIQ3IAXHZMA, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:47.308815677 (190h15m41.739895808s) [noise=true , address=104.36.82.38:4693 , piece_id=WDDDMKFR7SK4SGSGYQDMRJKXXOTRGGNT2QCEFOOA562WYD3DELNQ, seg_pos={0 0}, upload=197943]: Uploading piece... Aug 16 15:41:48.467916279 (190h15m42.89899641s) [upload=197943]: Next returned nil segment Aug 16 15:41:48.467933712 (190h15m42.899013843s) [upload=197943]: Waiting for error group managing segments... Aug 16 15:41:48.771827119 (190h15m43.20290725s) [noise=true , address=66.23.193.175:4559 , piece_id=BMQZYYHQS2BEO4SU6UIBU5EPD4MUPDRQMKPS26AS7BIQ3IAXHZMA, seg_pos={0 0}, upload=197943]: Done uploading piece. err: <nil> Aug 16 15:41:52.043663920 (190h15m46.474744051s) [noise=true , address=79.127.201.215:4345 , piece_id=D3FCD6NLHJD2YW75L546PV5ARE3KTEMOD2RTU4PKCLFYZANFB4AQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: <nil> Aug 16 15:41:55.721307829 (190h15m50.15238795s) [noise=true , address=45.91.116.12:4243 , piece_id=JHDTH5WCIV7ORESAAGNJJPESKYZRJ3WSPGVQSPXRMWIR6WMVIDMA, seg_pos={0 0}, upload=197943]: Done uploading piece. err: <nil> Aug 16 15:42:15.496750524 (190h16m9.927830655s) [noise=true , address=45.91.116.8:4072 , piece_id=TA4CVWYUYDINPJ7CAMFOO2BIOX5QRDB3ENDJ6RKOMAXQIZXMPVAQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: <nil> Aug 16 15:42:35.582008625 (190h16m30.013088746s) [noise=true , address=77.90.31.13:4649 , piece_id=LMCC6M7B2KNNMEVJGLX2PZUJ72QEI5J6GHY2L5VLG5IH5HYK4OLA, seg_pos={0 0}, upload=197943]: Done uploading piece. err: <nil> Aug 16 15:42:45.998072672 (190h16m40.429152803s) [noise=true , address=77.90.31.21:4902 , piece_id=CXXYON6ARXJQ7326EGNYV6IWUIEJB2Z6BMXC5P7LZCETM6O7EEMQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: expected piece hash; write tcp 188.166.225.170:37790->77.90.31.21:4902: use of closed network connection; write tcp 188.166.225.170:37790->77.90.31.21:4902: use of closed network connection; EOF Aug 16 15:42:45.998358590 (190h16m40.429438711s) [noise=true , address=104.36.82.19:4416 , piece_id=6VBLTHMANUA7ONGXKPI6OSTPKKK2XUZOEVT55TTT3L3W2NK323MQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12Rwbh1TjuvXvNEcTKsS49FMV3YKJUUzea9K8zFdDXns2ZyMcvg, address:104.36.82.19:4416): protocol: rpc: dial tcp 104.36.82.19:4416: i/o timeout; context deadline exceeded Aug 16 15:42:45.998459253 (190h16m40.429539384s) [noise=true , address=104.36.82.44:4880 , piece_id=JJVEVORNQA4FFDA22LSSKR23CLWKK6WUSE3BJNQIV7QF6W7W323Q, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998465465 (190h16m40.429545596s) [noise=true , address=104.36.82.23:4447 , piece_id=UGUT723Q2NQPIOSIKKRCEUURJQNZD2YEJISIP427ITELPHQFXJJQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998471416 (190h16m40.429551547s) [noise=true , address=104.36.82.40:4972 , piece_id=KWW3B4R4LI37B3EZNV446LORQRBA56SD7TPLROEY2O7NJICNTXGA, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:1XSuvCvq716GAa9WWmM3n2tqnfh8paaFUhjBzVzpiqwJtNifnr, address:104.36.82.40:4972): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998476605 (190h16m40.429556736s) [noise=true , address=104.36.82.38:4693 , piece_id=WDDDMKFR7SK4SGSGYQDMRJKXXOTRGGNT2QCEFOOA562WYD3DELNQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d, address:104.36.82.38:4693): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998524537 (190h16m40.429604668s) [noise=true , address=104.36.82.34:4554 , piece_id=IUPTOB2YWGCIXTO4PSPNV3ERVOI2O3XYFJHBQJNZKK2SWW2OBXKQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12rSDkPXm47w73hry9W6BWwAiwNpB3cJuwedNt1JuSEEtd3Sup7, address:104.36.82.34:4554): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998552321 (190h16m40.429632452s) [noise=true , address=104.36.82.60:4364 , piece_id=3P42GNO3VEN7MFTPUVURGJ7AB7EBJH6SMVK7JDH3QKYU2JLVJIZQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:104.36.82.60:4364): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998571137 (190h16m40.429651258s) [noise=true , address=104.36.82.2:4763 , piece_id=YHZ3OZ3DWPHIWECFQU3RTBXQS3J2NAWARUA5ZZVIMJWYNUV6OFTQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12P642GmVJzz139WqNE7AZFGieZUk1dNQPwEJ3RSeWVJ4xUZPuZ, address:104.36.82.2:4763): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998617906 (190h16m40.429698037s) [noise=true , address=104.36.82.35:4603 , piece_id=PJC3J4F74FKJRQZGVOYLZJWROETWZRVK255K3ISR4V2WQ2ELHTQQ, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998650207 (190h16m40.429730338s) [noise=true , address=104.36.82.5:4844 , piece_id=VQ4ZM5M3X7XF6FYYGA3TUJTOSQKYMF6DZGWUZFRCOAK3YR22JZ3Q, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998674855 (190h16m40.429754986s) [noise=true , address=104.36.82.59:4329 , piece_id=CCO5T2GLDXOPETVASOINP5UAE2TMX3DRQ7HWY6322RG2WVN4YFPA, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:1dGY6BG4FULjuRwnEKd9wV4cZPVJsgo32oFHKZn5BgjxVaojbk, address:104.36.82.59:4329): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998711145 (190h16m40.429791276s) [noise=true , address=104.36.82.3:4796 , piece_id=W2H4HO7T3RRHYAC4LLUR56MLRZPSWI7FQBMR5JTBTBCJBPTUWB4A, seg_pos={0 0}, upload=197943]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:42:45.998724971 (190h16m40.429805102s) [seg_pos={0 0}, upload=197943]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Aug 16 15:42:46.374199811 (190h16m40.805279942s) [upload=197943]: Done uploading segments... Aug 16 15:42:46.374224719 (190h16m40.80530485s) [upload=197943]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.377778326s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.234146879s FAIL (2025-08-16 15:42:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== oise=true , address=104.36.82.62:4434 , piece_id=EJSSIRWYU2HZ3UIIN4PJQY3H6Q3SLH5HTJV7D5W3O6GPBLR6PKEQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.227446786 (190h15m31.658526917s) [noise=true , address=104.36.82.44:4902 , piece_id=X5IIPJRNXEOWYOPJECMZQG6CN3L6VOJNHYK3TNTJEL46E4QFCPVQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.227765236 (190h15m31.658845367s) [noise=true , address=77.90.31.13:4658 , piece_id=2KLFS3TT2ZDDEKAOKGNJ7UAUAN7OQM3IHU2BD34UILWATZKL4L3Q, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.227853775 (190h15m31.658933906s) [noise=true , address=77.90.31.19:4853 , piece_id=EU573AGWE7U5KXUZZUM47RSWELO7GGM5ARMYBJACQM3C42IMSABQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.228029661 (190h15m31.659109792s) [noise=true , address=104.36.82.60:4361 , piece_id=L7NUJGDGYKWWWSV77ZBC67YO5APTU5WO4QLDNQCYB63EQRFF5JKA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.228133260 (190h15m31.659213391s) [noise=true , address=77.90.31.11:4584 , piece_id=VKYUBZGKUINCRQJCGFFZFKUCQ74IDYDFGMTPQQ7J2E6JQ575QJ6Q, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.228362398 (190h15m31.659442519s) [noise=true , address=104.36.82.59:4324 , piece_id=FB2AMPWZXWGKLMZPJNVDG2W6GX5Y4UXU7CAA45UG7VG5I2EZ4JCQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.228609421 (190h15m31.659689552s) [noise=true , address=77.90.31.21:4880 , piece_id=AHBIMAERDBXHMEIV7TE7YBSMGKNP3C6TMV5E36UAYHDN6Y3PYWHQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.228878987 (190h15m31.659959118s) [noise=true , address=77.90.31.14:4684 , piece_id=HXMK4OW7IZFLOL2ATVVX4YBTIIU7RDWYYMXJKBMRO3L3QQV2QYDA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.229258453 (190h15m31.660338584s) [noise=true , address=45.91.116.8:4072 , piece_id=4RYYS2CXQ6VP2AOQ3UZJLHMXP3AANL2ZBNVZECUHWIR6QH6RLYYA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.229506768 (190h15m31.660586889s) [noise=true , address=77.90.31.16:4757 , piece_id=RSQ4V3R5RDOETMTXANBGNFWTC2F3K6DSF3HARRZMWHYRA4A7UKYA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.229725827 (190h15m31.660805958s) [noise=true , address=79.127.201.215:4360 , piece_id=A5HEE5ADKDTH2U2MDFCKPV6QBMA74KFP2XNRAGU3F7ZAXWAIS3LQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.229973290 (190h15m31.661053421s) [noise=true , address=79.127.201.214:4322 , piece_id=GB6JUYTOCLBADRO6T2EPI6ALET4OGNC6UB4XOZ23DYQZLLXJI2SA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.230106195 (190h15m31.661186336s) [noise=true , address=77.90.31.12:4599 , piece_id=CGIVUEPR7KHJJXNOXCPAPLE4BELSD2XYIWAOCT66APSRR6TRZDSA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.230398174 (190h15m31.661478305s) [noise=true , address=79.127.220.97:4392 , piece_id=VUJJ4S2FWXIMP2HASH4KNJZJ6NIFVU6E3QJWBSGHDI5RC4G7ESOQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.230704730 (190h15m31.661784851s) [noise=true , address=104.36.82.23:4470 , piece_id=KQGDP4LSQQY4IBFAPJLPOXMDUR5XW3A4ZMO5LZOFSBW6G7YQDPGQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.231035904 (190h15m31.662116025s) [noise=true , address=79.127.220.101:4073 , piece_id=XDDVPKWASR6AQP2ALF7FA4CBTN2QB3LJA6JJHOAJ2LOXGVX72SBA, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:37.231112431 (190h15m31.662192562s) [noise=true , address=104.36.82.8:4989 , piece_id=JW732CHSF2LWF2ARZQXZE6SZMEQOAVI4EIB5CVEBV7ZUKZK5HKLQ, seg_pos={0 0}, upload=197940]: Uploading piece... Aug 16 15:41:38.702422160 (190h15m33.133502291s) [upload=197940]: Next returned nil segment Aug 16 15:41:38.702441245 (190h15m33.133521376s) [upload=197940]: Waiting for error group managing segments... Aug 16 15:41:39.085700259 (190h15m33.51678038s) [noise=true , address=79.127.220.97:4392 , piece_id=VUJJ4S2FWXIMP2HASH4KNJZJ6NIFVU6E3QJWBSGHDI5RC4G7ESOQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:41:39.854265083 (190h15m34.285345215s) [noise=true , address=79.127.201.214:4322 , piece_id=GB6JUYTOCLBADRO6T2EPI6ALET4OGNC6UB4XOZ23DYQZLLXJI2SA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:41:40.301422722 (190h15m34.732502843s) [noise=true , address=79.127.220.101:4073 , piece_id=XDDVPKWASR6AQP2ALF7FA4CBTN2QB3LJA6JJHOAJ2LOXGVX72SBA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:41:50.190304902 (190h15m44.621385033s) [noise=true , address=79.127.201.215:4360 , piece_id=A5HEE5ADKDTH2U2MDFCKPV6QBMA74KFP2XNRAGU3F7ZAXWAIS3LQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:41:56.527279228 (190h15m50.958359359s) [noise=true , address=77.90.31.16:4757 , piece_id=RSQ4V3R5RDOETMTXANBGNFWTC2F3K6DSF3HARRZMWHYRA4A7UKYA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:41:56.764731677 (190h15m51.195811798s) [noise=true , address=77.90.31.11:4584 , piece_id=VKYUBZGKUINCRQJCGFFZFKUCQ74IDYDFGMTPQQ7J2E6JQ575QJ6Q, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:02.633699272 (190h15m57.064779423s) [noise=true , address=45.91.116.8:4072 , piece_id=4RYYS2CXQ6VP2AOQ3UZJLHMXP3AANL2ZBNVZECUHWIR6QH6RLYYA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:03.992491182 (190h15m58.423571323s) [noise=true , address=77.90.31.12:4599 , piece_id=CGIVUEPR7KHJJXNOXCPAPLE4BELSD2XYIWAOCT66APSRR6TRZDSA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:09.744462570 (190h16m4.175542701s) [noise=true , address=77.90.31.14:4684 , piece_id=HXMK4OW7IZFLOL2ATVVX4YBTIIU7RDWYYMXJKBMRO3L3QQV2QYDA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:20.078768424 (190h16m14.509848615s) [noise=true , address=77.90.31.19:4853 , piece_id=EU573AGWE7U5KXUZZUM47RSWELO7GGM5ARMYBJACQM3C42IMSABQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:26.108657396 (190h16m20.539737527s) [noise=true , address=77.90.31.13:4658 , piece_id=2KLFS3TT2ZDDEKAOKGNJ7UAUAN7OQM3IHU2BD34UILWATZKL4L3Q, seg_pos={0 0}, upload=197940]: Done uploading piece. err: <nil> Aug 16 15:42:36.119741388 (190h16m30.550821519s) [noise=true , address=104.36.82.59:4324 , piece_id=FB2AMPWZXWGKLMZPJNVDG2W6GX5Y4UXU7CAA45UG7VG5I2EZ4JCQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:12j5y2JKTy6UuY5iC2MLsNNXkSKzNhj8BQ4ZwDtV3Nz4wjfGzkH, address:104.36.82.59:4324): protocol: write tcp 188.166.225.170:48202->104.36.82.59:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.119793578 (190h16m30.550873709s) [noise=true , address=104.36.82.44:4902 , piece_id=X5IIPJRNXEOWYOPJECMZQG6CN3L6VOJNHYK3TNTJEL46E4QFCPVQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: write tcp 188.166.225.170:52580->104.36.82.44:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.119827081 (190h16m30.550907202s) [noise=true , address=104.36.82.60:4361 , piece_id=L7NUJGDGYKWWWSV77ZBC67YO5APTU5WO4QLDNQCYB63EQRFF5JKA, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: write tcp 188.166.225.170:44190->104.36.82.60:4361: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.119861126 (190h16m30.550941257s) [noise=true , address=77.90.31.21:4880 , piece_id=AHBIMAERDBXHMEIV7TE7YBSMGKNP3C6TMV5E36UAYHDN6Y3PYWHQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, address:77.90.31.21:4880): protocol: write tcp 188.166.225.170:38106->77.90.31.21:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.119914659 (190h16m30.55099479s) [noise=true , address=104.36.82.62:4434 , piece_id=EJSSIRWYU2HZ3UIIN4PJQY3H6Q3SLH5HTJV7D5W3O6GPBLR6PKEQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:12kZH2amRgBmyujKY7Kk4yEQRY8CJMfaz2hw2XRJ8tzRiHExvtX, address:104.36.82.62:4434): protocol: rpc: dial tcp 104.36.82.62:4434: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.119971768 (190h16m30.551051899s) [noise=true , address=104.36.82.8:4989 , piece_id=JW732CHSF2LWF2ARZQXZE6SZMEQOAVI4EIB5CVEBV7ZUKZK5HKLQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:14QoLDSHxoQN2FBG8w5kQTxUvtpvnGNZf5qgJGpGB43eTBJ511, address:104.36.82.8:4989): protocol: write tcp 188.166.225.170:57516->104.36.82.8:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.120005472 (190h16m30.551085603s) [noise=true , address=104.36.82.23:4470 , piece_id=KQGDP4LSQQY4IBFAPJLPOXMDUR5XW3A4ZMO5LZOFSBW6G7YQDPGQ, seg_pos={0 0}, upload=197940]: Done uploading piece. err: ecclient: upload failed (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF, address:104.36.82.23:4470): protocol: rpc: dial tcp 104.36.82.23:4470: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:36.120082089 (190h16m30.55116223s) [seg_pos={0 0}, upload=197940]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 16 15:42:36.352264905 (190h16m30.783345036s) [upload=197940]: Done uploading segments... Aug 16 15:42:36.352331582 (190h16m30.783411713s) [upload=197940]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.234158592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [...X...X...X..................X.X..X.....X..................] duration: 3.026102566s FAIL (2025-08-16 15:42:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 3.026125841s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................................X......X..XXXXXXXXXX.XX] duration: 30.22853505s FAIL (2025-08-16 15:42:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:41:45.100124180 (190h15m39.531204311s) [upload=197941]: Starting upload Aug 16 15:41:45.100231585 (190h15m39.531311716s) [upload=197941]: Done starting upload Aug 16 15:41:45.100352096 (190h15m39.531432227s) [upload=197941]: Uploading segments... Aug 16 15:41:45.100588929 (190h15m39.53166906s) [upload=197941]: Got next segment. Inline: false Aug 16 15:41:45.100630899 (190h15m39.53171102s) [upload=197941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:45.100636049 (190h15m39.53171618s) [upload=197941]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:45.354341927 (190h15m39.785422058s) [seg_pos={0 0}, upload=197941]: Begin upload segment... Aug 16 15:41:45.354398555 (190h15m39.785478676s) [seg_pos={0 0}, upload=197941]: Done begin upload segment. Aug 16 15:41:45.354426589 (190h15m39.78550672s) [noise=true , address=104.36.82.11:4114 , piece_id=IIE6IDPCC5AAQ2IX5L3LJSACNUWW6SA7D6VADS7ZU5P4RARGBWEQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.354451016 (190h15m39.785531147s) [noise=true , address=45.91.116.4:4937 , piece_id=QNNFWJ2HXISOFXN6FGMW2NE5KLDFHWKU7I7LBKBE6CBROAWB75ZQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.354822868 (190h15m39.785902999s) [noise=true , address=104.36.82.11:4115 , piece_id=IGLJ4UP4UDPSGWLKB43BMAEUYUSSB3TOY5TBGV2TWIOL42DZC6MQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355005598 (190h15m39.786085719s) [noise=true , address=104.36.82.15:4259 , piece_id=DEOBBLNEZNPP3HA7L4FZPB4ONPPBWWYGBZLCRAJYJD3GKUXQEJVQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355132720 (190h15m39.786212851s) [noise=true , address=104.36.82.7:4909 , piece_id=WPSE2E6WSD4ZIUKP7PYZPOJQAN2CJYAKJLUZ3KQX342TV6MGH36A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355301553 (190h15m39.786381684s) [noise=true , address=77.90.31.13:4628 , piece_id=HRGBGIN6GWHP5333NQ5XDJQONCJVFJW2CIMVEVBT3VTOBZRPXW4A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355432855 (190h15m39.786512976s) [noise=true , address=104.36.82.8:4969 , piece_id=6FNXQ3W5WMBFVDKOC3AMLUB7NGUK45UCIXQNZGR4US3JONH7H3HA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.355747998 (190h15m39.786828129s) [noise=true , address=77.90.31.10:4518 , piece_id=QHFSQLVPFURVJ6PLYPNPHYEMN7EYRBTCKE5OQ6IQWULRAAVFV5DQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356065455 (190h15m39.787145586s) [noise=true , address=45.91.116.8:4047 , piece_id=SDZQ2IV3CB4W2KDJ3PXZWSV77IDB7IHLEZTFHHRU6OSTX32Q7JTA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356324761 (190h15m39.787404892s) [noise=true , address=104.36.82.7:4912 , piece_id=TKXZVAHQJM5RVW6OD7VAL2RZIX6Z5XS5TS2XM76CCSP3I6QL4NLQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356562967 (190h15m39.787643098s) [noise=true , address=104.36.82.12:4151 , piece_id=7H5WSMJPW2Q676WNHCITVIRF3K6T745ZSCRW2QTEAQ4SQU2KDQRA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.356787115 (190h15m39.787867247s) [noise=true , address=104.36.82.6:4874 , piece_id=IIVXF7WUEDCRYCTRBQTKM4FJROW4U6WKGDVA7BAGR2ILRF2Q4ABQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357044529 (190h15m39.78812466s) [noise=true , address=104.36.82.10:4079 , piece_id=HIAOJGHNSH6L434KH4MDOTHT52XOOKPQE2ZW3KNB3ENW7KPGFGNQ, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357263728 (190h15m39.788343859s) [noise=true , address=45.91.116.12:4227 , piece_id=RIH4WNWQJYQLXW4COZSCQJG7EDVBTZPFTIJ3EKHYUT4T3M66IVAA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357499458 (190h15m39.78857959s) [noise=true , address=104.36.82.13:4187 , piece_id=YS2JY5V7FRLNURFGFPWAOOR6ZFIVGVJ6PI7C2DIRSAHLEI6F545A, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:45.357729038 (190h15m39.788809159s) [noise=true , address=77.90.31.16:4735 , piece_id=K2IGNDSGY7PXM4SMMDB4OPFRTIFTIW4IJJSDGPCGDIMLDKALHFZA, seg_pos={0 0}, upload=197941]: Uploading piece... Aug 16 15:41:47.119119724 (190h15m41.550199855s) [upload=197941]: Next returned nil segment Aug 16 15:41:47.119137288 (190h15m41.550217419s) [upload=197941]: Waiting for error group managing segments... Aug 16 15:41:47.402462310 (190h15m41.833542431s) [noise=true , address=104.36.82.7:4909 , piece_id=WPSE2E6WSD4ZIUKP7PYZPOJQAN2CJYAKJLUZ3KQX342TV6MGH36A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.419585042 (190h15m41.850665163s) [noise=true , address=104.36.82.10:4079 , piece_id=HIAOJGHNSH6L434KH4MDOTHT52XOOKPQE2ZW3KNB3ENW7KPGFGNQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.489994573 (190h15m41.921074714s) [noise=true , address=104.36.82.12:4151 , piece_id=7H5WSMJPW2Q676WNHCITVIRF3K6T745ZSCRW2QTEAQ4SQU2KDQRA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.503742047 (190h15m41.934822178s) [noise=true , address=104.36.82.15:4259 , piece_id=DEOBBLNEZNPP3HA7L4FZPB4ONPPBWWYGBZLCRAJYJD3GKUXQEJVQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:47.521485137 (190h15m41.952565268s) [noise=true , address=104.36.82.6:4874 , piece_id=IIVXF7WUEDCRYCTRBQTKM4FJROW4U6WKGDVA7BAGR2ILRF2Q4ABQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:49.572924071 (190h15m44.004004202s) [noise=true , address=104.36.82.7:4912 , piece_id=TKXZVAHQJM5RVW6OD7VAL2RZIX6Z5XS5TS2XM76CCSP3I6QL4NLQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:54.705143870 (190h15m49.136224002s) [noise=true , address=45.91.116.4:4937 , piece_id=QNNFWJ2HXISOFXN6FGMW2NE5KLDFHWKU7I7LBKBE6CBROAWB75ZQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:56.697020750 (190h15m51.128100871s) [noise=true , address=45.91.116.12:4227 , piece_id=RIH4WNWQJYQLXW4COZSCQJG7EDVBTZPFTIJ3EKHYUT4T3M66IVAA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:41:57.897503371 (190h15m52.328583503s) [noise=true , address=77.90.31.16:4735 , piece_id=K2IGNDSGY7PXM4SMMDB4OPFRTIFTIW4IJJSDGPCGDIMLDKALHFZA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:42:02.088178699 (190h15m56.51925883s) [noise=true , address=45.91.116.8:4047 , piece_id=SDZQ2IV3CB4W2KDJ3PXZWSV77IDB7IHLEZTFHHRU6OSTX32Q7JTA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: <nil> Aug 16 15:42:14.433410600 (190h16m8.864490721s) [noise=true , address=77.90.31.13:4628 , piece_id=HRGBGIN6GWHP5333NQ5XDJQONCJVFJW2CIMVEVBT3VTOBZRPXW4A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:42:14.433608378 (190h16m8.864688509s) [noise=true , address=77.90.31.10:4518 , piece_id=QHFSQLVPFURVJ6PLYPNPHYEMN7EYRBTCKE5OQ6IQWULRAAVFV5DQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433646671 (190h16m8.864726802s) [noise=true , address=104.36.82.11:4114 , piece_id=IIE6IDPCC5AAQ2IX5L3LJSACNUWW6SA7D6VADS7ZU5P4RARGBWEQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:44480->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433688491 (190h16m8.864768612s) [noise=true , address=104.36.82.11:4115 , piece_id=IGLJ4UP4UDPSGWLKB43BMAEUYUSSB3TOY5TBGV2TWIOL42DZC6MQ, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433716796 (190h16m8.864796927s) [noise=true , address=104.36.82.8:4969 , piece_id=6FNXQ3W5WMBFVDKOC3AMLUB7NGUK45UCIXQNZGR4US3JONH7H3HA, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433728437 (190h16m8.864808568s) [noise=true , address=104.36.82.13:4187 , piece_id=YS2JY5V7FRLNURFGFPWAOOR6ZFIVGVJ6PI7C2DIRSAHLEI6F545A, seg_pos={0 0}, upload=197941]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:42:14.433747454 (190h16m8.864827585s) [seg_pos={0 0}, upload=197941]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:42:14.659856879 (190h16m9.09093702s) [upload=197941]: Done uploading segments... Aug 16 15:42:14.659898739 (190h16m9.090978901s) [upload=197941]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.228565519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........X......X...X.X......X..X......X..X...XXXXXXXXXXXXX] duration: 1m0.529330705s FAIL (2025-08-16 15:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:40:45.866625988 (190h14m40.297706119s) [upload=197929]: Starting upload Aug 16 15:40:45.866674049 (190h14m40.29775418s) [upload=197929]: Done starting upload Aug 16 15:40:45.866717643 (190h14m40.297797764s) [upload=197929]: Uploading segments... Aug 16 15:40:45.866844536 (190h14m40.297924657s) [upload=197929]: Got next segment. Inline: false Aug 16 15:40:45.866867619 (190h14m40.297947751s) [upload=197929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:40:45.866871618 (190h14m40.297951749s) [upload=197929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:40:46.410061241 (190h14m40.841141373s) [seg_pos={0 0}, upload=197929]: Begin upload segment... Aug 16 15:40:46.410106599 (190h14m40.84118673s) [seg_pos={0 0}, upload=197929]: Done begin upload segment. Aug 16 15:40:46.410130164 (190h14m40.841210295s) [noise=true , address=77.90.31.21:4882 , piece_id=JPOYL2T5BCOKJFJ5SFMZTMVD2MU2DYU6ARNBA3I73WLYC2GXEW3A, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.410255233 (190h14m40.841335394s) [noise=true , address=45.91.116.5:4314 , piece_id=PEH22OQ2M54XCE2XUCGOPMTKHWMNTAMXNPQOPKCLTISI27EJBJEQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.410483850 (190h14m40.841563981s) [noise=true , address=45.91.116.10:4090 , piece_id=FSDXDVQOHT3NXJHFC5WXIFGBF5CW5VXFWHDSMMBKS36ALDZ4DG4Q, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.410766261 (190h14m40.841846392s) [noise=true , address=104.36.82.59:4318 , piece_id=3OAWNFU7FLNJ2FC2HWMLMWKCFJAUUMUJCZK6VS2O64QAIZAZ7YFA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.410949601 (190h14m40.842029752s) [noise=true , address=104.36.82.60:4354 , piece_id=YACVAVB7JQCA5TPRNAATLUVSSDO3SWQ6EYJFLDAGPSS2DJZWP3SQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411056265 (190h14m40.842136396s) [noise=true , address=104.36.82.17:4359 , piece_id=CBCRKWXSPBLYFDHPH7A6U62XCJVLDULYV65VRH5RUJNMZPSJHGUA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411327274 (190h14m40.842407405s) [noise=true , address=79.127.220.101:4071 , piece_id=P4YJLBAN6HGE5FTC57MN6OQV34NJX6VNS6NVECKWYQIIMYGKJHFQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411439138 (190h14m40.842519269s) [noise=true , address=104.36.82.35:4616 , piece_id=JAAUUJKXBFCLNJINFMZU4YKT5DKZI4CKCKKLEJ3XGA2S324I42VA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411632838 (190h14m40.842712969s) [noise=true , address=79.127.201.214:4320 , piece_id=34RAKUZKBVTDFAPYV2HSHLY2FGPQTJ6IYXIZLXBJIFCOZWDIM7FQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411840416 (190h14m40.842920547s) [noise=true , address=77.90.31.15:4706 , piece_id=JIGPCQDLEQHG7S57EUC5YLOZ46TBRFLLKLMVCK5DS7IUTYNPT73A, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.411898026 (190h14m40.842978157s) [noise=true , address=77.90.31.19:4845 , piece_id=WAVZNKHPV2WBDCCVWNELQCZVNCCT73AD3AKI6SGGLIANQCTY45BQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.412126082 (190h14m40.843206213s) [noise=true , address=66.23.193.175:4553 , piece_id=M4DDSCFO6KLLE53YI7JUFDQABOAEUY6XAQ3FIW2ZBLLAIQVUBSKA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.412201567 (190h14m40.843281698s) [noise=true , address=104.36.82.23:4473 , piece_id=WGQS6A5BWD7BTLXWLLKDTBQPJW4C5PDMHPON5LEZPUWELLGIL6BA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.412364338 (190h14m40.843444469s) [noise=true , address=104.36.82.5:4864 , piece_id=X5L556SV7VHHNWDSHCYESCRO4ELPWQN73EQDXYD3ZCJ4H6SBTONA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.412575963 (190h14m40.843656084s) [noise=true , address=45.91.116.6:4985 , piece_id=3OOVXIIUAFF6FIIKZ7GH7Y5SU5VYLZNFAYNRW44CWZX2RFKPZYLQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.412611411 (190h14m40.843691552s) [noise=true , address=104.36.82.37:4653 , piece_id=PXSEGAV6ZRIQTGKMIKLO3A24ESX2GDPKFPPBHRAEJCZAUG4A43XA, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.413000816 (190h14m40.844080947s) [noise=true , address=104.36.82.2:4741 , piece_id=TTWJADXNDYBQW544GGL4O72AY53CBNVZHIO3SV7JJTCVWYKLSDFQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:46.413360965 (190h14m40.844441086s) [noise=true , address=104.36.82.58:4298 , piece_id=DHPPEID32RI5XH7AKIEHXHFVY4JN6QX32Q7LZVVPZIZDA3NQGAXQ, seg_pos={0 0}, upload=197929]: Uploading piece... Aug 16 15:40:47.512251493 (190h14m41.943331624s) [upload=197929]: Next returned nil segment Aug 16 15:40:47.512267052 (190h14m41.943347183s) [upload=197929]: Waiting for error group managing segments... Aug 16 15:40:48.503843595 (190h14m42.934923726s) [noise=true , address=104.36.82.5:4864 , piece_id=X5L556SV7VHHNWDSHCYESCRO4ELPWQN73EQDXYD3ZCJ4H6SBTONA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:40:48.971825156 (190h14m43.402905287s) [noise=true , address=79.127.220.101:4071 , piece_id=P4YJLBAN6HGE5FTC57MN6OQV34NJX6VNS6NVECKWYQIIMYGKJHFQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:40:49.518989546 (190h14m43.950069677s) [noise=true , address=79.127.201.214:4320 , piece_id=34RAKUZKBVTDFAPYV2HSHLY2FGPQTJ6IYXIZLXBJIFCOZWDIM7FQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:40:50.598846574 (190h14m45.029926745s) [noise=true , address=66.23.193.175:4553 , piece_id=M4DDSCFO6KLLE53YI7JUFDQABOAEUY6XAQ3FIW2ZBLLAIQVUBSKA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:02.699384392 (190h14m57.130464543s) [noise=true , address=45.91.116.10:4090 , piece_id=FSDXDVQOHT3NXJHFC5WXIFGBF5CW5VXFWHDSMMBKS36ALDZ4DG4Q, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:10.772750484 (190h15m5.203830605s) [noise=true , address=45.91.116.5:4314 , piece_id=PEH22OQ2M54XCE2XUCGOPMTKHWMNTAMXNPQOPKCLTISI27EJBJEQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:12.354506210 (190h15m6.785586341s) [noise=true , address=45.91.116.6:4985 , piece_id=3OOVXIIUAFF6FIIKZ7GH7Y5SU5VYLZNFAYNRW44CWZX2RFKPZYLQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:21.524771168 (190h15m15.955851299s) [noise=true , address=77.90.31.19:4845 , piece_id=WAVZNKHPV2WBDCCVWNELQCZVNCCT73AD3AKI6SGGLIANQCTY45BQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:29.280467811 (190h15m23.711547942s) [noise=true , address=77.90.31.15:4706 , piece_id=JIGPCQDLEQHG7S57EUC5YLOZ46TBRFLLKLMVCK5DS7IUTYNPT73A, seg_pos={0 0}, upload=197929]: Done uploading piece. err: <nil> Aug 16 15:41:45.219983426 (190h15m39.651063557s) [noise=true , address=77.90.31.21:4882 , piece_id=JPOYL2T5BCOKJFJ5SFMZTMVD2MU2DYU6ARNBA3I73WLYC2GXEW3A, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 188.166.225.170:47758->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:47758->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47758->77.90.31.21:4882: use of closed network connection Aug 16 15:41:45.220032040 (190h15m39.651112171s) [noise=true , address=104.36.82.17:4359 , piece_id=CBCRKWXSPBLYFDHPH7A6U62XCJVLDULYV65VRH5RUJNMZPSJHGUA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1MdZDBkhh7vwffrHksdiixu99PiTgmmNAzdUo39YGaEtpuAHdR, address:104.36.82.17:4359): protocol: rpc: dial tcp 104.36.82.17:4359: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220074672 (190h15m39.651154793s) [noise=true , address=104.36.82.35:4616 , piece_id=JAAUUJKXBFCLNJINFMZU4YKT5DKZI4CKCKKLEJ3XGA2S324I42VA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1rDCZdLm3zP15D7MoanzkotJ4HJjvj1GS1R9aJdQmDJpvEboin, address:104.36.82.35:4616): protocol: rpc: dial tcp 104.36.82.35:4616: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220104008 (190h15m39.651184139s) [noise=true , address=104.36.82.58:4298 , piece_id=DHPPEID32RI5XH7AKIEHXHFVY4JN6QX32Q7LZVVPZIZDA3NQGAXQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1MWsyGhKkDhjngUDDRvxmhQddbNnRZ1A3mpzEtjLs9ZcRC5LBz, address:104.36.82.58:4298): protocol: rpc: dial tcp 104.36.82.58:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220142311 (190h15m39.651222442s) [noise=true , address=104.36.82.2:4741 , piece_id=TTWJADXNDYBQW544GGL4O72AY53CBNVZHIO3SV7JJTCVWYKLSDFQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:12kbyfvxZKx55ADddbca6MvotTA6pCrQeTws8xhPKoSTQ1GpYzK, address:104.36.82.2:4741): protocol: write tcp 188.166.225.170:51648->104.36.82.2:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220164373 (190h15m39.651244504s) [noise=true , address=104.36.82.59:4318 , piece_id=3OAWNFU7FLNJ2FC2HWMLMWKCFJAUUMUJCZK6VS2O64QAIZAZ7YFA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:12abkQbak5f8FneYvCZo5FVi4RNZZfijQZFDd4Y1aacLUNgLnyM, address:104.36.82.59:4318): protocol: write tcp 188.166.225.170:51192->104.36.82.59:4318: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220195643 (190h15m39.651275774s) [noise=true , address=104.36.82.60:4354 , piece_id=YACVAVB7JQCA5TPRNAATLUVSSDO3SWQ6EYJFLDAGPSS2DJZWP3SQ, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv, address:104.36.82.60:4354): protocol: rpc: dial tcp 104.36.82.60:4354: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220203879 (190h15m39.65128401s) [noise=true , address=104.36.82.23:4473 , piece_id=WGQS6A5BWD7BTLXWLLKDTBQPJW4C5PDMHPON5LEZPUWELLGIL6BA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220219508 (190h15m39.65129964s) [noise=true , address=104.36.82.37:4653 , piece_id=PXSEGAV6ZRIQTGKMIKLO3A24ESX2GDPKFPPBHRAEJCZAUG4A43XA, seg_pos={0 0}, upload=197929]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:104.36.82.37:4653): protocol: rpc: dial tcp 104.36.82.37:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:45.220245278 (190h15m39.651325409s) [seg_pos={0 0}, upload=197929]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:41:45.747250895 (190h15m40.178331016s) [upload=197929]: Done uploading segments... Aug 16 15:41:45.747276975 (190h15m40.178357096s) [upload=197929]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.529354449s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.235100905s FAIL (2025-08-16 15:41:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 7 , piece_id=N3B7D5POFFKOAYT5K2YMLMXYH3FKOLT5CMAAMRI7JBJ73CHV6AGA, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.747713103 (190h14m31.178793224s) [noise=true , address=104.36.82.18:4384 , piece_id=L6FOGNZWMAFBPXC2OKCNU2NRPUI5ML5QQAMDJB63URQANUOKMWDQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.747957541 (190h14m31.179037672s) [noise=true , address=104.36.82.8:4993 , piece_id=AU63QDTUNDCNIDT7QZS3YWQINPMVITJOYYMBUHNJ35GERBZT5X4Q, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.748202109 (190h14m31.17928224s) [noise=true , address=104.36.82.35:4596 , piece_id=MWDI2JA7FBGZYAK4WHJ6KQ3ZRMENX4YVDP7TQF2YNYNRYN6MRGUQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.748501020 (190h14m31.179581151s) [noise=true , address=104.36.82.23:4473 , piece_id=2XPHYZGROQQANMITLZSZBV4XEMOQG4PKLXFX63SYSB7AV4S54Z3A, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.748748695 (190h14m31.179828826s) [noise=true , address=45.91.116.9:4270 , piece_id=K2TYIQO3JUOIGJJVB2JJAO5Z6PXVWHOHXN5PUOLV7CC6HGRJGCSQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.748987341 (190h14m31.180067472s) [noise=true , address=104.36.82.45:4934 , piece_id=2O5IINQ7VLRLTSBZSSE5K5H5TOGEIRKBHUFP5TT6LAAH5YBNXT3A, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.749208224 (190h14m31.180288355s) [noise=true , address=77.90.31.19:4874 , piece_id=SU2XZM5YGRRJP7KGWVY35XEL3AG434ECSV7VFY4TZVA7U2U5BGQQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.749439557 (190h14m31.180519688s) [noise=true , address=77.90.31.10:4532 , piece_id=5LT7M3WEOYPS2SPAGDLV37DF4A37BLQ4UNE2XG3KI32T4M7POCVQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.749686339 (190h14m31.18076647s) [noise=true , address=104.36.82.16:4306 , piece_id=WUHTD4F3PMJMUAPPGGOQSPGMOCJ5HBFYIT6OTPOZQ5ULVZEWNQJQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.749995580 (190h14m31.181075711s) [noise=true , address=45.91.116.7:4018 , piece_id=D66VOGVXO6BG3HJEGFE3BW3LAGXFEYLTWQ7BFEBIBTZY5C6AZTFA, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.750241912 (190h14m31.181322043s) [noise=true , address=77.90.31.21:4889 , piece_id=RJ2NGITQS4PAPZV4SCYHOJKLJNDXFLW3YOR5YY3VKG3HYP5GZRXQ, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.750492110 (190h14m31.181572231s) [noise=true , address=104.36.82.58:4270 , piece_id=U6WUXXSAP2RRLBGK3HYJPIOPORHGTPZ4TZBIVLBXVCJQ2OKQ3PTA, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:36.750710027 (190h14m31.181790148s) [noise=true , address=104.36.82.44:4896 , piece_id=FP2SNOJQ7HRONKCCXYYY7LUTGM6TK3CPNQNX3IRKDDFEUEJXH6BA, seg_pos={0 0}, upload=197925]: Uploading piece... Aug 16 15:40:37.961153740 (190h14m32.392233871s) [upload=197925]: Next returned nil segment Aug 16 15:40:37.961171954 (190h14m32.392252085s) [upload=197925]: Waiting for error group managing segments... Aug 16 15:40:40.808588055 (190h14m35.239668176s) [noise=true , address=79.127.201.215:4360 , piece_id=R6BLCL4IZNBM6ZWPR4RBVTJ4W4XLBIF6MO2FIPXOVO4IYTDSDJRA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:40:44.793228630 (190h14m39.224308771s) [noise=true , address=45.91.116.7:4018 , piece_id=D66VOGVXO6BG3HJEGFE3BW3LAGXFEYLTWQ7BFEBIBTZY5C6AZTFA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: storage node overloaded, request limit: 2000 Aug 16 15:40:46.424881088 (190h14m40.855961219s) [noise=true , address=66.23.193.175:4559 , piece_id=X3545GNA2JXJTC6NX7WKMJDZZVMQJRDMMIGRUF5G5CJ53V3KMEZQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:40:47.821407358 (190h14m42.252487529s) [noise=true , address=104.36.82.5:4847 , piece_id=N3B7D5POFFKOAYT5K2YMLMXYH3FKOLT5CMAAMRI7JBJ73CHV6AGA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:40:53.084158377 (190h14m47.515238508s) [noise=true , address=77.90.31.19:4874 , piece_id=SU2XZM5YGRRJP7KGWVY35XEL3AG434ECSV7VFY4TZVA7U2U5BGQQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:40:55.050054125 (190h14m49.481134256s) [noise=true , address=104.36.82.45:4934 , piece_id=2O5IINQ7VLRLTSBZSSE5K5H5TOGEIRKBHUFP5TT6LAAH5YBNXT3A, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:40:58.094602486 (190h14m52.525682607s) [noise=true , address=45.91.116.9:4270 , piece_id=K2TYIQO3JUOIGJJVB2JJAO5Z6PXVWHOHXN5PUOLV7CC6HGRJGCSQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:41:10.948094355 (190h15m5.379174486s) [noise=true , address=104.36.82.58:4270 , piece_id=U6WUXXSAP2RRLBGK3HYJPIOPORHGTPZ4TZBIVLBXVCJQ2OKQ3PTA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: <nil> Aug 16 15:41:35.651796501 (190h15m30.082876632s) [noise=true , address=77.90.31.14:4664 , piece_id=WRCVBJCXGLB6PAQNFMTHUZXI7S6R2LKWGMH4P5L6XOI7UF2Z64SQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:41:35.652023014 (190h15m30.083103145s) [noise=true , address=104.36.82.8:4993 , piece_id=AU63QDTUNDCNIDT7QZS3YWQINPMVITJOYYMBUHNJ35GERBZT5X4Q, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1xnx9shwwZAcWVV6aPj2ikukj8nbKiFsBs1hiS89zSkRQqNMJD, address:104.36.82.8:4993): protocol: rpc: dial tcp 104.36.82.8:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652045828 (190h15m30.083125959s) [noise=true , address=104.36.82.16:4306 , piece_id=WUHTD4F3PMJMUAPPGGOQSPGMOCJ5HBFYIT6OTPOZQ5ULVZEWNQJQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652082658 (190h15m30.083162789s) [noise=true , address=104.36.82.38:4695 , piece_id=5H2ZIK37ZUIMJSG636HGLBI5WBLGFTXF3233UYYG2QGUXQSCTVAA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y, address:104.36.82.38:4695): protocol: rpc: dial tcp 104.36.82.38:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652115551 (190h15m30.083195672s) [noise=true , address=104.36.82.23:4473 , piece_id=2XPHYZGROQQANMITLZSZBV4XEMOQG4PKLXFX63SYSB7AV4S54Z3A, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652150678 (190h15m30.083230799s) [noise=true , address=104.36.82.44:4896 , piece_id=FP2SNOJQ7HRONKCCXYYY7LUTGM6TK3CPNQNX3IRKDDFEUEJXH6BA, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: rpc: dial tcp 104.36.82.44:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652201425 (190h15m30.083281556s) [noise=true , address=77.90.31.10:4532 , piece_id=5LT7M3WEOYPS2SPAGDLV37DF4A37BLQ4UNE2XG3KI32T4M7POCVQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: read tcp 188.166.225.170:46638->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652218208 (190h15m30.083298339s) [noise=true , address=104.36.82.18:4384 , piece_id=L6FOGNZWMAFBPXC2OKCNU2NRPUI5ML5QQAMDJB63URQANUOKMWDQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:12hwbJ9oH6qJD6rucYWqk6BZStAtpM6GY98kC94Tb1edJbpxARE, address:104.36.82.18:4384): protocol: write tcp 188.166.225.170:56032->104.36.82.18:4384: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652276339 (190h15m30.08335647s) [noise=true , address=77.90.31.21:4889 , piece_id=RJ2NGITQS4PAPZV4SCYHOJKLJNDXFLW3YOR5YY3VKG3HYP5GZRXQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: write tcp 188.166.225.170:52766->77.90.31.21:4889: use of closed network connection; write tcp 188.166.225.170:52766->77.90.31.21:4889: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52766->77.90.31.21:4889: use of closed network connection Aug 16 15:41:35.652326225 (190h15m30.083406356s) [noise=true , address=104.36.82.35:4596 , piece_id=MWDI2JA7FBGZYAK4WHJ6KQ3ZRMENX4YVDP7TQF2YNYNRYN6MRGUQ, seg_pos={0 0}, upload=197925]: Done uploading piece. err: ecclient: upload failed (node:11HDJFxgZxCiibSDmX5sdwZ1jSEd2NpBPBjMQ858nmRiMGfP7q, address:104.36.82.35:4596): protocol: write tcp 188.166.225.170:58856->104.36.82.35:4596: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:35.652352595 (190h15m30.083432726s) [seg_pos={0 0}, upload=197925]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:41:35.884321141 (190h15m30.315401272s) [upload=197925]: Done uploading segments... Aug 16 15:41:35.884349737 (190h15m30.315429868s) [upload=197925]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.235119119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [....X...X...X..................X.X..X.....X.................] duration: 2.882010222s FAIL (2025-08-16 15:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.88203485s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................................X......X..XXXXXXXXXX.X] duration: 30.23094044s FAIL (2025-08-16 15:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== g_pos={0 0}, upload=197924]: Done begin upload segment. Aug 16 15:40:34.274177384 (190h14m28.705257505s) [noise=true , address=104.36.82.5:4840 , piece_id=5VO3W37R4MHI6NBG2JXNON2RDRIJLSQ2VGH3Y62HFVVRGZLL6FBQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.274195990 (190h14m28.705276121s) [noise=true , address=45.91.116.7:4009 , piece_id=DLHVAYU4GH75GEGIOE3AWIHVFV67ZPXWHGPRBEZPEUKMXZG4IY7A, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.274558874 (190h14m28.705639005s) [noise=true , address=45.91.116.8:4047 , piece_id=7RBPAI6Y5VWT7V7SXRKJKFNFQ3Q4E32EXZC73ZQBIUE6YKEOGBCQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.274565126 (190h14m28.705645257s) [noise=true , address=104.36.82.5:4837 , piece_id=TTACTDON7VGGUHX5WK3FY3VEMVAODULLU6SN74T35WQYAV7B2SWA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.274814714 (190h14m28.705894845s) [noise=true , address=104.36.82.15:4258 , piece_id=AQKH4AWIEOFOCDYDTA26CQV7S5JLOCKTUQMWNBOPZTERKSAJKDBA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.274844952 (190h14m28.705925123s) [noise=true , address=77.90.31.10:4517 , piece_id=7RP5XJVNCGBRLEP55WOCGXJFHF6EIES2IRKLIIENGY3NCCPYFWVA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275106632 (190h14m28.706186763s) [noise=true , address=104.36.82.10:4077 , piece_id=KWIER2GIABDAANK7FVX4W3KQD2IHQF4BWK7PJMKCAD4LCD4YUFPQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275148022 (190h14m28.706228153s) [noise=true , address=104.36.82.11:4115 , piece_id=DB6TT2BSOVCWPWDLGZF7Q2SUE3INCLSEMMQ3JBIBMPNAUDUUYA2Q, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275354797 (190h14m28.706434928s) [noise=true , address=77.90.31.10:4518 , piece_id=J3EGLZGAMGCVEPIF4ZRJWPPSRURZTSBIFULKQNAC2U33UI22EEUA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275379054 (190h14m28.706459185s) [noise=true , address=77.90.31.21:4878 , piece_id=BTLM54RUO3UFOK4O6IHOSSVMTK7UNWGUT46G5TLLPJVBB5YN52HQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275606839 (190h14m28.70668697s) [noise=true , address=45.91.116.4:4938 , piece_id=K6T6N66ZPL5NSJBBF3Q44HVLKTKPHDZXVGLFUFPQH423M53FNSEQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275755133 (190h14m28.706835264s) [noise=true , address=104.36.82.7:4909 , piece_id=5TXHIB33TJWS3I6VNOWMDKXECVTQDD2KTQDYCLYSHFTHUL4XAGUA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275838202 (190h14m28.706918333s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ3LCYIGF6ILAVDK7U3TQX4TPJYZ66HU5NW67PBQ275MNG7CKJSA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.275982418 (190h14m28.707062549s) [noise=true , address=45.91.116.12:4227 , piece_id=DD7DCF46H5BFXT3QIOEF3YCHW6WSQPM22X3YAK6OIIO7EFVHMSYQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.276262904 (190h14m28.707343035s) [noise=true , address=104.36.82.6:4873 , piece_id=YJO4OMKQKBKEM4OIISUNX7GQPY3C2QRD2UD4GUNTRKZEWFUW2ZIQ, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:34.276936684 (190h14m28.708016815s) [noise=true , address=104.36.82.12:4149 , piece_id=7UDDEM632JFSL5BN3T2PXIYZMNNCJFCI2KE6Q7733VAJAHLKUJBA, seg_pos={0 0}, upload=197924]: Uploading piece... Aug 16 15:40:35.980420533 (190h14m30.411500664s) [upload=197924]: Next returned nil segment Aug 16 15:40:35.980436673 (190h14m30.411516804s) [upload=197924]: Waiting for error group managing segments... Aug 16 15:40:36.327304658 (190h14m30.758384779s) [noise=true , address=104.36.82.6:4873 , piece_id=YJO4OMKQKBKEM4OIISUNX7GQPY3C2QRD2UD4GUNTRKZEWFUW2ZIQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:36.379892298 (190h14m30.810972419s) [noise=true , address=104.36.82.10:4077 , piece_id=KWIER2GIABDAANK7FVX4W3KQD2IHQF4BWK7PJMKCAD4LCD4YUFPQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:36.429664742 (190h14m30.860744873s) [noise=true , address=104.36.82.7:4909 , piece_id=5TXHIB33TJWS3I6VNOWMDKXECVTQDD2KTQDYCLYSHFTHUL4XAGUA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:36.434860565 (190h14m30.865940696s) [noise=true , address=104.36.82.15:4258 , piece_id=AQKH4AWIEOFOCDYDTA26CQV7S5JLOCKTUQMWNBOPZTERKSAJKDBA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:39.447670297 (190h14m33.878750428s) [noise=true , address=104.36.82.12:4149 , piece_id=7UDDEM632JFSL5BN3T2PXIYZMNNCJFCI2KE6Q7733VAJAHLKUJBA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:48.281131089 (190h14m42.71221122s) [noise=true , address=104.36.82.5:4837 , piece_id=TTACTDON7VGGUHX5WK3FY3VEMVAODULLU6SN74T35WQYAV7B2SWA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:48.430884516 (190h14m42.861964667s) [noise=true , address=104.36.82.5:4840 , piece_id=5VO3W37R4MHI6NBG2JXNON2RDRIJLSQ2VGH3Y62HFVVRGZLL6FBQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:40:55.705281220 (190h14m50.136361351s) [noise=true , address=45.91.116.4:4938 , piece_id=K6T6N66ZPL5NSJBBF3Q44HVLKTKPHDZXVGLFUFPQH423M53FNSEQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: <nil> Aug 16 15:41:03.356957776 (190h14m57.788037898s) [noise=true , address=45.91.116.12:4227 , piece_id=DD7DCF46H5BFXT3QIOEF3YCHW6WSQPM22X3YAK6OIIO7EFVHMSYQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:59800->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:59800->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59800->45.91.116.12:4227: use of closed network connection Aug 16 15:41:03.357140096 (190h14m57.788220217s) [noise=true , address=45.91.116.8:4047 , piece_id=7RBPAI6Y5VWT7V7SXRKJKFNFQ3Q4E32EXZC73ZQBIUE6YKEOGBCQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:47362->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:47362->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47362->45.91.116.8:4047: use of closed network connection Aug 16 15:41:03.357198828 (190h14m57.788278959s) [noise=true , address=77.90.31.10:4518 , piece_id=J3EGLZGAMGCVEPIF4ZRJWPPSRURZTSBIFULKQNAC2U33UI22EEUA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:03.357212814 (190h14m57.788292955s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ3LCYIGF6ILAVDK7U3TQX4TPJYZ66HU5NW67PBQ275MNG7CKJSA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:41:03.357249916 (190h14m57.788330047s) [noise=true , address=104.36.82.11:4115 , piece_id=DB6TT2BSOVCWPWDLGZF7Q2SUE3INCLSEMMQ3JBIBMPNAUDUUYA2Q, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:43536->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:03.357285724 (190h14m57.788365855s) [noise=true , address=77.90.31.21:4878 , piece_id=BTLM54RUO3UFOK4O6IHOSSVMTK7UNWGUT46G5TLLPJVBB5YN52HQ, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59330->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59330->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59330->77.90.31.21:4878: use of closed network connection Aug 16 15:41:03.357324438 (190h14m57.788404569s) [noise=true , address=45.91.116.7:4009 , piece_id=DLHVAYU4GH75GEGIOE3AWIHVFV67ZPXWHGPRBEZPEUKMXZG4IY7A, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:56422->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:56422->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56422->45.91.116.7:4009: use of closed network connection Aug 16 15:41:03.357358233 (190h14m57.788438364s) [noise=true , address=77.90.31.10:4517 , piece_id=7RP5XJVNCGBRLEP55WOCGXJFHF6EIES2IRKLIIENGY3NCCPYFWVA, seg_pos={0 0}, upload=197924]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:40100->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:03.357376487 (190h14m57.788456618s) [seg_pos={0 0}, upload=197924]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:41:03.586555936 (190h14m58.017636067s) [upload=197924]: Done uploading segments... Aug 16 15:41:03.586585323 (190h14m58.017665454s) [upload=197924]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.230960748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [...........X......X...X.X......X..X......X..X...XXXXXXXXXXXX] duration: 1m0.258885947s FAIL (2025-08-16 15:40:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:39:44.793441450 (190h13m39.224521572s) [upload=197912]: Starting upload Aug 16 15:39:44.793493340 (190h13m39.224573471s) [upload=197912]: Done starting upload Aug 16 15:39:44.793580577 (190h13m39.224660709s) [upload=197912]: Uploading segments... Aug 16 15:39:44.793670559 (190h13m39.224750691s) [upload=197912]: Got next segment. Inline: false Aug 16 15:39:44.793704795 (190h13m39.224784926s) [upload=197912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:39:44.793708882 (190h13m39.224789013s) [upload=197912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:39:45.301185533 (190h13m39.732265664s) [seg_pos={0 0}, upload=197912]: Begin upload segment... Aug 16 15:39:45.301230398 (190h13m39.732310519s) [seg_pos={0 0}, upload=197912]: Done begin upload segment. Aug 16 15:39:45.301254725 (190h13m39.732334856s) [noise=true , address=45.91.116.6:4999 , piece_id=XBSIKEOA6VU6YIWLRQ7FZIX3DSW67PXNTZ64LYPPUOMPGTSB4LEQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.301323297 (190h13m39.732403428s) [noise=true , address=45.91.116.12:4244 , piece_id=OTBOU4AVANDWU4QQJW4VS4WZBKB5CI5XE6OHJ2SSQBF6HYNPBMFQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.301637958 (190h13m39.732718089s) [noise=true , address=104.36.82.38:4705 , piece_id=F46XI3MCQFV3JMGTXE6COFT3KOBO2CAN5V4JMWLQIOJ5XFXHCOHQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.301868310 (190h13m39.732948441s) [noise=true , address=104.36.82.36:4635 , piece_id=HJFCN3XMPXOU6ANGLXQYUF4MKZIF63NRSHRHICBVLGLGYY7WNYJQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.301980584 (190h13m39.733060715s) [noise=true , address=104.36.82.44:4888 , piece_id=WYLOY33U5MQOTHDBVZJ7JITS7NS3ENJV6JABJLRUQMYVWBHUJSAA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.302180757 (190h13m39.733260878s) [noise=true , address=79.127.201.215:4356 , piece_id=YD5JX3QSI6YXJYJ6I65GLNJZLCX4W34KAM2C6EGEKT65DUTLHBJA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.302307229 (190h13m39.733387361s) [noise=true , address=77.90.31.12:4604 , piece_id=5NH73FFORA6J4MOXXDULMC7QDHMRNBBN5V2HNPO4XR23BL6LUVIQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.302494467 (190h13m39.733574598s) [noise=true , address=104.36.82.3:4780 , piece_id=CQ6PWNLAQMF77CHE24KRGCJY6EKWS6EPX3POKLBKH6YUY6VULX3Q, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.302603586 (190h13m39.733683717s) [noise=true , address=104.36.82.34:4557 , piece_id=F3JY2AJ7VVZKCSDSHUTVA56LFPGU4ITQYOSXI45JIMS4FKPNZWWQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.302812336 (190h13m39.733892457s) [noise=true , address=77.90.31.19:4862 , piece_id=L3257TCJ6GWP4BZONKGJS3DQALA7CRGDIMV6TYNHZQWRS7XJYSGA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303000225 (190h13m39.734080356s) [noise=true , address=104.36.82.5:4860 , piece_id=HXOCPPJPTYCFCYBAHRQHLRHYV3XV3ZD2W4W7Y2TUWLHZC2I2JNCQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303100226 (190h13m39.734180357s) [noise=true , address=104.36.82.58:4297 , piece_id=T5XQMWP4UP2DZSFI4IAUQNZMJOE5VDROQWW5OXGXRK2M6XWQILCA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303254401 (190h13m39.734334532s) [noise=true , address=77.90.31.18:4816 , piece_id=TRORXP3DBXOETEVSBFUYXPZSW4YDQX4ZSETOHZ6QMI7UEJVXT4BA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303358952 (190h13m39.734439083s) [noise=true , address=104.36.82.2:4748 , piece_id=4BQHQDRANEO4H7QVIKVV2WLWN4B25UQAS646NMQQ34G4BPIFLB2A, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303484802 (190h13m39.734564933s) [noise=true , address=77.90.31.11:4587 , piece_id=OEY5VEL24B7EF6MG3RVP6QJNIJ6TECWTKAEYMP3MERYAOAORZ6EQ, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303593299 (190h13m39.73467343s) [noise=true , address=66.23.193.175:4553 , piece_id=V7ECRBLJWVENKEXA6KWILOFYV7XDWQXJKIUWZLMB5GLSZZLVETVA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303741333 (190h13m39.734821464s) [noise=true , address=104.36.82.23:4464 , piece_id=VNGVR2S3Z7AXBNHNMTI4H34RCIEMF6RUM4ME7SE3F5Y2FKBOT6IA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:45.303977084 (190h13m39.735057205s) [noise=true , address=77.90.31.16:4745 , piece_id=NNJ5M546ONV2MV6JBZ7HHXL2XY2FXA7KEOTRMJFGF4JX5JOGFMPA, seg_pos={0 0}, upload=197912]: Uploading piece... Aug 16 15:39:46.513038070 (190h13m40.944118201s) [upload=197912]: Next returned nil segment Aug 16 15:39:46.513058719 (190h13m40.94413885s) [upload=197912]: Waiting for error group managing segments... Aug 16 15:39:48.184311494 (190h13m42.615391645s) [noise=true , address=79.127.201.215:4356 , piece_id=YD5JX3QSI6YXJYJ6I65GLNJZLCX4W34KAM2C6EGEKT65DUTLHBJA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:49.440238283 (190h13m43.871318414s) [noise=true , address=77.90.31.11:4587 , piece_id=OEY5VEL24B7EF6MG3RVP6QJNIJ6TECWTKAEYMP3MERYAOAORZ6EQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:49.599915758 (190h13m44.030995889s) [noise=true , address=77.90.31.12:4604 , piece_id=5NH73FFORA6J4MOXXDULMC7QDHMRNBBN5V2HNPO4XR23BL6LUVIQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:50.341176910 (190h13m44.772257041s) [noise=true , address=77.90.31.16:4745 , piece_id=NNJ5M546ONV2MV6JBZ7HHXL2XY2FXA7KEOTRMJFGF4JX5JOGFMPA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:50.441045418 (190h13m44.872125539s) [noise=true , address=77.90.31.18:4816 , piece_id=TRORXP3DBXOETEVSBFUYXPZSW4YDQX4ZSETOHZ6QMI7UEJVXT4BA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:52.822671688 (190h13m47.253751819s) [noise=true , address=77.90.31.19:4862 , piece_id=L3257TCJ6GWP4BZONKGJS3DQALA7CRGDIMV6TYNHZQWRS7XJYSGA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:54.195854483 (190h13m48.626934614s) [noise=true , address=66.23.193.175:4553 , piece_id=V7ECRBLJWVENKEXA6KWILOFYV7XDWQXJKIUWZLMB5GLSZZLVETVA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:54.998656618 (190h13m49.429736749s) [noise=true , address=45.91.116.12:4244 , piece_id=OTBOU4AVANDWU4QQJW4VS4WZBKB5CI5XE6OHJ2SSQBF6HYNPBMFQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:39:55.193213020 (190h13m49.624293151s) [noise=true , address=45.91.116.6:4999 , piece_id=XBSIKEOA6VU6YIWLRQ7FZIX3DSW67PXNTZ64LYPPUOMPGTSB4LEQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: <nil> Aug 16 15:40:44.141695031 (190h14m38.572775162s) [noise=true , address=104.36.82.44:4888 , piece_id=WYLOY33U5MQOTHDBVZJ7JITS7NS3ENJV6JABJLRUQMYVWBHUJSAA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.141793690 (190h14m38.572900142s) [noise=true , address=104.36.82.34:4557 , piece_id=F3JY2AJ7VVZKCSDSHUTVA56LFPGU4ITQYOSXI45JIMS4FKPNZWWQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT, address:104.36.82.34:4557): protocol: rpc: dial tcp 104.36.82.34:4557: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.141857051 (190h14m38.572937182s) [noise=true , address=104.36.82.5:4860 , piece_id=HXOCPPJPTYCFCYBAHRQHLRHYV3XV3ZD2W4W7Y2TUWLHZC2I2JNCQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.141886818 (190h14m38.572966949s) [noise=true , address=104.36.82.38:4705 , piece_id=F46XI3MCQFV3JMGTXE6COFT3KOBO2CAN5V4JMWLQIOJ5XFXHCOHQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:12kt12GvRGL3iNLVzYSgvs3Ha2Tzf2FApGVsK9aeuNMsQcfGUBw, address:104.36.82.38:4705): protocol: write tcp 188.166.225.170:38830->104.36.82.38:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.141944669 (190h14m38.5730248s) [noise=true , address=104.36.82.36:4635 , piece_id=HJFCN3XMPXOU6ANGLXQYUF4MKZIF63NRSHRHICBVLGLGYY7WNYJQ, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:104.36.82.36:4635): protocol: rpc: dial tcp 104.36.82.36:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.141978884 (190h14m38.573059005s) [noise=true , address=104.36.82.58:4297 , piece_id=T5XQMWP4UP2DZSFI4IAUQNZMJOE5VDROQWW5OXGXRK2M6XWQILCA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:12fVAGmLTkEZ2oauRnaMgyV9vvbRKZHj6eKW5vf2yZbPMDpRwfW, address:104.36.82.58:4297): protocol: rpc: dial tcp 104.36.82.58:4297: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.142029211 (190h14m38.573109342s) [noise=true , address=104.36.82.23:4464 , piece_id=VNGVR2S3Z7AXBNHNMTI4H34RCIEMF6RUM4ME7SE3F5Y2FKBOT6IA, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: rpc: dial tcp 104.36.82.23:4464: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.142156454 (190h14m38.573236585s) [noise=true , address=104.36.82.2:4748 , piece_id=4BQHQDRANEO4H7QVIKVV2WLWN4B25UQAS646NMQQ34G4BPIFLB2A, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:104.36.82.2:4748): protocol: write tcp 188.166.225.170:36922->104.36.82.2:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.142163738 (190h14m38.573243889s) [noise=true , address=104.36.82.3:4780 , piece_id=CQ6PWNLAQMF77CHE24KRGCJY6EKWS6EPX3POKLBKH6YUY6VULX3Q, seg_pos={0 0}, upload=197912]: Done uploading piece. err: ecclient: upload failed (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM, address:104.36.82.3:4780): protocol: rpc: dial tcp 104.36.82.3:4780: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:44.142240615 (190h14m38.573320746s) [seg_pos={0 0}, upload=197912]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:40:44.398847314 (190h14m38.829927435s) [upload=197912]: Done uploading segments... Aug 16 15:40:44.398870719 (190h14m38.82995085s) [upload=197912]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.258898351s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.232351893s FAIL (2025-08-16 15:40:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 15:39:36.290354085 (190h13m30.721434276s) [noise=true , address=45.91.116.9:4270 , piece_id=OG4JWJPGNTIOBTKLWNXCKWIQI2OYKJTXA6YL63FXLC3YA6ACRV3Q, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.290535021 (190h13m30.721615192s) [noise=true , address=77.90.31.13:4648 , piece_id=26X72AOJV4U7F5ZHHFQ2C5VPR35FO4YZJ4O5HQ62APH3V4RMHU4Q, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.290994320 (190h13m30.722074461s) [noise=true , address=104.36.82.35:4591 , piece_id=EBD6TLRQOL5MTNCTUTQQ35OTBWYL7TUT3D56SQ3QRSQ3X2RV767Q, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.291216795 (190h13m30.722296956s) [noise=true , address=104.36.82.38:4718 , piece_id=RAQKS5TZ7PYJPJROF4LEYPC2T3OKEIIA2R6ZWDIAUQ6TALPUEFWA, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.291469989 (190h13m30.72255013s) [noise=true , address=104.36.82.58:4286 , piece_id=EXQIYTEMOMTYEPBHYHLNHB5WVBHAYIQGZGKYDEA3G2QE732SFIZQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.291787818 (190h13m30.722867969s) [noise=true , address=104.36.82.2:4744 , piece_id=KWQ54MM52EL4XN56EODGT6CLVRTVFCF2CHNDN4XHSXOYJX75DLIA, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.291898490 (190h13m30.722978621s) [noise=true , address=104.36.82.44:4877 , piece_id=SXG54CB7VTX44HWZGYSBCKO7VVJGKNZ3PSLXGTVRGNGMPLEZT3GA, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.292340085 (190h13m30.723420216s) [noise=true , address=104.36.82.23:4447 , piece_id=AUD6KBEE5RPTMSGIG3FMPWMDXHT7XPIO4NQGX5VMTDDV5MJIWNPQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.292345374 (190h13m30.723425516s) [noise=true , address=104.36.82.62:4438 , piece_id=4WYH6K5DO5D35CJM7FJEHNCQFVPCBJNOEH26VHN6LJRGS3UAS4EQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.292381244 (190h13m30.723461375s) [noise=true , address=104.36.82.60:4346 , piece_id=PIAFNVB2Q3DM2CQZHCDXK6YVIYFT6MJOQFJSLCSJ6H34R533XAFA, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.292742024 (190h13m30.723822155s) [noise=true , address=45.91.116.10:4090 , piece_id=ILF52CGSEI2GGI7GNCD5RIMVBHOQAI5O2TAN5TX4ASPPSST6F6GQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.292881731 (190h13m30.723961862s) [noise=true , address=77.90.31.11:4563 , piece_id=QX7YLCATXSIPX57JK2JGAWGG2OSWQT4FLYKKSHJ55A7O6C6XAAZQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.293069040 (190h13m30.724149181s) [noise=true , address=104.36.82.39:4729 , piece_id=MVBTQKO4CGJ6BJ5JULYRLGDS6UGV4FZJNDYP43DCOYA6EHJIVVIQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.293285122 (190h13m30.724365253s) [noise=true , address=79.127.201.215:4359 , piece_id=FZ4JZXWE367XVKQHLNUIEMVYFS6FOLFRC7CLDGFLZFN736NGXG4A, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.293387839 (190h13m30.72446798s) [noise=true , address=77.90.31.12:4604 , piece_id=THEOXPWTD76ZTTS5ET75O4GPSSTSGDPGBQKM7I5NFWOTO7F5GKUQ, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:36.293697162 (190h13m30.724777303s) [noise=true , address=66.23.193.175:4564 , piece_id=7J6VE5NM2G4ZUGEWJMS5OD3KVQV6JP6CU2NAWMN5RHLLMQCBUHXA, seg_pos={0 0}, upload=197907]: Uploading piece... Aug 16 15:39:37.481852557 (190h13m31.912932688s) [upload=197907]: Next returned nil segment Aug 16 15:39:37.481870772 (190h13m31.912950903s) [upload=197907]: Waiting for error group managing segments... Aug 16 15:39:39.176261113 (190h13m33.607341244s) [noise=true , address=77.90.31.13:4648 , piece_id=26X72AOJV4U7F5ZHHFQ2C5VPR35FO4YZJ4O5HQ62APH3V4RMHU4Q, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:39.321900869 (190h13m33.75298099s) [noise=true , address=77.90.31.12:4604 , piece_id=THEOXPWTD76ZTTS5ET75O4GPSSTSGDPGBQKM7I5NFWOTO7F5GKUQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:39.334954816 (190h13m33.766034947s) [noise=true , address=77.90.31.11:4563 , piece_id=QX7YLCATXSIPX57JK2JGAWGG2OSWQT4FLYKKSHJ55A7O6C6XAAZQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:39.540333515 (190h13m33.971413646s) [noise=true , address=45.91.116.7:4017 , piece_id=CP3XUQU2EWU2ANORWDXF5AJQQH6JK4UJ43FPZPCONPKF3V7O5EHA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:39.564620595 (190h13m33.995700726s) [noise=true , address=45.91.116.9:4270 , piece_id=OG4JWJPGNTIOBTKLWNXCKWIQI2OYKJTXA6YL63FXLC3YA6ACRV3Q, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:39.896074244 (190h13m34.327154375s) [noise=true , address=45.91.116.7:4043 , piece_id=QVS2X4EAIPJQH5KMFOV7DHHXKIDI2ECBASNZIOCHXZRD3UQM7DXQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:40.634399860 (190h13m35.065479991s) [noise=true , address=45.91.116.10:4090 , piece_id=ILF52CGSEI2GGI7GNCD5RIMVBHOQAI5O2TAN5TX4ASPPSST6F6GQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:45.901887881 (190h13m40.332968012s) [noise=true , address=66.23.193.175:4564 , piece_id=7J6VE5NM2G4ZUGEWJMS5OD3KVQV6JP6CU2NAWMN5RHLLMQCBUHXA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:39:55.287438238 (190h13m49.718518369s) [noise=true , address=79.127.201.215:4359 , piece_id=FZ4JZXWE367XVKQHLNUIEMVYFS6FOLFRC7CLDGFLZFN736NGXG4A, seg_pos={0 0}, upload=197907]: Done uploading piece. err: <nil> Aug 16 15:40:35.177432463 (190h14m29.608512594s) [noise=true , address=104.36.82.62:4438 , piece_id=4WYH6K5DO5D35CJM7FJEHNCQFVPCBJNOEH26VHN6LJRGS3UAS4EQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:1r45NCqiic7hJHEm5f2ho7TvCkT15cgGVHMKqxuJZThDH4Eqq9, address:104.36.82.62:4438): protocol: rpc: dial tcp 104.36.82.62:4438: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177779948 (190h14m29.608860079s) [noise=true , address=104.36.82.2:4744 , piece_id=KWQ54MM52EL4XN56EODGT6CLVRTVFCF2CHNDN4XHSXOYJX75DLIA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: write tcp 188.166.225.170:40122->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177859089 (190h14m29.60893922s) [noise=true , address=104.36.82.38:4718 , piece_id=RAQKS5TZ7PYJPJROF4LEYPC2T3OKEIIA2R6ZWDIAUQ6TALPUEFWA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98, address:104.36.82.38:4718): protocol: rpc: dial tcp 104.36.82.38:4718: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177874819 (190h14m29.60895495s) [noise=true , address=104.36.82.58:4286 , piece_id=EXQIYTEMOMTYEPBHYHLNHB5WVBHAYIQGZGKYDEA3G2QE732SFIZQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:104.36.82.58:4286): protocol: rpc: dial tcp 104.36.82.58:4286: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177901370 (190h14m29.608981501s) [noise=true , address=104.36.82.60:4346 , piece_id=PIAFNVB2Q3DM2CQZHCDXK6YVIYFT6MJOQFJSLCSJ6H34R533XAFA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:104.36.82.60:4346): protocol: rpc: dial tcp 104.36.82.60:4346: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177911339 (190h14m29.608991471s) [noise=true , address=104.36.82.35:4591 , piece_id=EBD6TLRQOL5MTNCTUTQQ35OTBWYL7TUT3D56SQ3QRSQ3X2RV767Q, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55, address:104.36.82.35:4591): protocol: rpc: dial tcp 104.36.82.35:4591: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177936969 (190h14m29.6090171s) [noise=true , address=104.36.82.23:4447 , piece_id=AUD6KBEE5RPTMSGIG3FMPWMDXHT7XPIO4NQGX5VMTDDV5MJIWNPQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: rpc: dial tcp 104.36.82.23:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177965112 (190h14m29.609045243s) [noise=true , address=104.36.82.44:4877 , piece_id=SXG54CB7VTX44HWZGYSBCKO7VVJGKNZ3PSLXGTVRGNGMPLEZT3GA, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: write tcp 188.166.225.170:50160->104.36.82.44:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177971414 (190h14m29.609051545s) [noise=true , address=104.36.82.39:4729 , piece_id=MVBTQKO4CGJ6BJ5JULYRLGDS6UGV4FZJNDYP43DCOYA6EHJIVVIQ, seg_pos={0 0}, upload=197907]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:104.36.82.39:4729): protocol: write tcp 188.166.225.170:39700->104.36.82.39:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:40:35.177986402 (190h14m29.609066533s) [seg_pos={0 0}, upload=197907]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:40:35.408103516 (190h14m29.839183647s) [upload=197907]: Done uploading segments... Aug 16 15:40:35.408129987 (190h14m29.839210118s) [upload=197907]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.232363606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......................................X......X..XXXXXXXXXX.] duration: 23.811392201s FAIL (2025-08-16 15:40:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:39:25.418649058 (190h13m19.849729189s) [upload=197906]: Starting upload Aug 16 15:39:25.418705837 (190h13m19.849785968s) [upload=197906]: Done starting upload Aug 16 15:39:25.418884368 (190h13m19.849964489s) [upload=197906]: Uploading segments... Aug 16 15:39:25.418915948 (190h13m19.849996069s) [upload=197906]: Got next segment. Inline: false Aug 16 15:39:25.418937299 (190h13m19.85001743s) [upload=197906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:39:25.418947058 (190h13m19.850027189s) [upload=197906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:39:25.656712546 (190h13m20.087792677s) [seg_pos={0 0}, upload=197906]: Begin upload segment... Aug 16 15:39:25.656747704 (190h13m20.087827835s) [seg_pos={0 0}, upload=197906]: Done begin upload segment. Aug 16 15:39:25.656764626 (190h13m20.087844747s) [noise=true , address=104.36.82.8:4969 , piece_id=VWTIRBFPGJYQ45QU5HDOQSJVE2RKRRHH7OEHVHTBNCBH2Q4V65AQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.656780987 (190h13m20.087861118s) [noise=true , address=104.36.82.9:4007 , piece_id=JHIJNYORGPFFUCLRPFD3WR2LYT43ETZAVIOI5GVM4GPCI45JKN4A, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657110027 (190h13m20.088190158s) [noise=true , address=104.36.82.10:4079 , piece_id=X5BJHBJBPUYVTESEHTO4GDCSW2RL3IIPRAQYNHK3KSMVCN75PPHQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657115227 (190h13m20.088195358s) [noise=true , address=45.91.116.5:4297 , piece_id=WKO7WTAHEQM3QLNF5TPXEADX73HNP7HDQG3PXHNCIMJAODRT6HYA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657401574 (190h13m20.088481705s) [noise=true , address=45.91.116.12:4225 , piece_id=LYJANRVYRY3I47OFGYEE23U45M6SFABJ25VGPDYT237UX2G7OGEQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657454726 (190h13m20.088534857s) [noise=true , address=45.91.116.8:4047 , piece_id=RJC7VBLWQG3IWDT26WBHTU5FBBSYOG53AH6ZO3TFUGYHYXALDDFA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657817942 (190h13m20.088898063s) [noise=true , address=104.36.82.14:4221 , piece_id=FZFP6Y74VTJSUD2YEQF6FBLTI2CWFYNB2JF6VYZPRCCBN3D2EM3Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658124448 (190h13m20.089204579s) [noise=true , address=45.91.116.13:4153 , piece_id=ZWRR7D73LCDRFOQ3QLVNOUQ2KBUDEEQFGD5YQ5Y7Y75IYIOCYXKQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658413160 (190h13m20.089493291s) [noise=true , address=77.90.31.18:4805 , piece_id=UH7UO5HDGUEG7X6TUZB44CF42B7P5VBOAMKOYBHED6WUSA2RJGDA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658680882 (190h13m20.089761013s) [noise=true , address=104.36.82.15:4259 , piece_id=FWIBYGGTKQ6E4ABW2C7YYMBRCA67MSXGCKZYBSLPLRNUL7ES4CTQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658942383 (190h13m20.090022514s) [noise=true , address=104.36.82.7:4909 , piece_id=PR6W2X3DVXB3P7WVI7SHQTBZX5X3MBFFHA5F332LGM2WT2CI3SKA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659148467 (190h13m20.090228598s) [noise=true , address=77.90.31.21:4877 , piece_id=34ACLNZFPGPOT7S7L3OVFKJVWMTALBDO22GDSMSB6TEK2HZRQR4Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659344572 (190h13m20.090424703s) [noise=true , address=104.36.82.15:4258 , piece_id=6H3A5SEBLV33R2QZA3QWSHEE4ABVWBQY2EEKPF6HRNSDTP4LBC7Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659551488 (190h13m20.090631619s) [noise=true , address=104.36.82.4:4801 , piece_id=B5N63JGP5CZQIH6SDKWX5LYUWGV5VQ6DZNS25VI7QFBZFZEV3BMQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659782891 (190h13m20.090863012s) [noise=true , address=77.90.31.13:4626 , piece_id=IU2ID4RHQMDDTMUGXI73TJ6CVTIH6WN4PISAPKLJOIZXT44TRWRA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.662332639 (190h13m20.09341277s) [noise=true , address=104.36.82.11:4113 , piece_id=OGQYTORMGCPWWUA2LS3WI3IVETZLW3B6AT2D3OJT4SDS2YA5YTXA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:27.442171003 (190h13m21.873251145s) [upload=197906]: Next returned nil segment Aug 16 15:39:27.442208184 (190h13m21.873288335s) [upload=197906]: Waiting for error group managing segments... Aug 16 15:39:27.725845033 (190h13m22.156925154s) [noise=true , address=104.36.82.15:4259 , piece_id=FWIBYGGTKQ6E4ABW2C7YYMBRCA67MSXGCKZYBSLPLRNUL7ES4CTQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.859612854 (190h13m22.290692985s) [noise=true , address=104.36.82.7:4909 , piece_id=PR6W2X3DVXB3P7WVI7SHQTBZX5X3MBFFHA5F332LGM2WT2CI3SKA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.860933421 (190h13m22.292013542s) [noise=true , address=104.36.82.10:4079 , piece_id=X5BJHBJBPUYVTESEHTO4GDCSW2RL3IIPRAQYNHK3KSMVCN75PPHQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.876058220 (190h13m22.307138341s) [noise=true , address=104.36.82.11:4113 , piece_id=OGQYTORMGCPWWUA2LS3WI3IVETZLW3B6AT2D3OJT4SDS2YA5YTXA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.935779070 (190h13m22.366859191s) [noise=true , address=104.36.82.15:4258 , piece_id=6H3A5SEBLV33R2QZA3QWSHEE4ABVWBQY2EEKPF6HRNSDTP4LBC7Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:28.519964965 (190h13m22.951045096s) [noise=true , address=45.91.116.5:4297 , piece_id=WKO7WTAHEQM3QLNF5TPXEADX73HNP7HDQG3PXHNCIMJAODRT6HYA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:28.759695192 (190h13m23.190775323s) [noise=true , address=77.90.31.13:4626 , piece_id=IU2ID4RHQMDDTMUGXI73TJ6CVTIH6WN4PISAPKLJOIZXT44TRWRA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.019266463 (190h13m23.450346584s) [noise=true , address=45.91.116.8:4047 , piece_id=RJC7VBLWQG3IWDT26WBHTU5FBBSYOG53AH6ZO3TFUGYHYXALDDFA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.062913397 (190h13m23.493993528s) [noise=true , address=77.90.31.18:4805 , piece_id=UH7UO5HDGUEG7X6TUZB44CF42B7P5VBOAMKOYBHED6WUSA2RJGDA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.076663566 (190h13m23.507743688s) [noise=true , address=45.91.116.12:4225 , piece_id=LYJANRVYRY3I47OFGYEE23U45M6SFABJ25VGPDYT237UX2G7OGEQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:31.421888487 (190h13m25.852968628s) [noise=true , address=45.91.116.13:4153 , piece_id=ZWRR7D73LCDRFOQ3QLVNOUQ2KBUDEEQFGD5YQ5Y7Y75IYIOCYXKQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:54.210295384 (190h13m48.641375515s) [noise=true , address=77.90.31.21:4877 , piece_id=34ACLNZFPGPOT7S7L3OVFKJVWMTALBDO22GDSMSB6TEK2HZRQR4Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:54.764506344 (190h13m49.195586475s) [noise=true , address=104.36.82.9:4007 , piece_id=JHIJNYORGPFFUCLRPFD3WR2LYT43ETZAVIOI5GVM4GPCI45JKN4A, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764576648 (190h13m49.195656779s) [noise=true , address=104.36.82.8:4969 , piece_id=VWTIRBFPGJYQ45QU5HDOQSJVE2RKRRHH7OEHVHTBNCBH2Q4V65AQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764601335 (190h13m49.195681466s) [noise=true , address=104.36.82.4:4801 , piece_id=B5N63JGP5CZQIH6SDKWX5LYUWGV5VQ6DZNS25VI7QFBZFZEV3BMQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:47940->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764659647 (190h13m49.195739768s) [noise=true , address=104.36.82.14:4221 , piece_id=FZFP6Y74VTJSUD2YEQF6FBLTI2CWFYNB2JF6VYZPRCCBN3D2EM3Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:34480->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764683572 (190h13m49.195763693s) [seg_pos={0 0}, upload=197906]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:39:55.006122002 (190h13m49.437202123s) [upload=197906]: Done uploading segments... Aug 16 15:39:55.006141679 (190h13m49.43722181s) [upload=197906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.811406019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....X...X...X..................X.X..X.....X................] duration: 1.870933757s FAIL (2025-08-16 15:40:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 1.870966419s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........................................X......X..XXXXXXXXXX] duration: 30.243615s FAIL (2025-08-16 15:39:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:39:25.418649058 (190h13m19.849729189s) [upload=197906]: Starting upload Aug 16 15:39:25.418705837 (190h13m19.849785968s) [upload=197906]: Done starting upload Aug 16 15:39:25.418884368 (190h13m19.849964489s) [upload=197906]: Uploading segments... Aug 16 15:39:25.418915948 (190h13m19.849996069s) [upload=197906]: Got next segment. Inline: false Aug 16 15:39:25.418937299 (190h13m19.85001743s) [upload=197906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:39:25.418947058 (190h13m19.850027189s) [upload=197906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:39:25.656712546 (190h13m20.087792677s) [seg_pos={0 0}, upload=197906]: Begin upload segment... Aug 16 15:39:25.656747704 (190h13m20.087827835s) [seg_pos={0 0}, upload=197906]: Done begin upload segment. Aug 16 15:39:25.656764626 (190h13m20.087844747s) [noise=true , address=104.36.82.8:4969 , piece_id=VWTIRBFPGJYQ45QU5HDOQSJVE2RKRRHH7OEHVHTBNCBH2Q4V65AQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.656780987 (190h13m20.087861118s) [noise=true , address=104.36.82.9:4007 , piece_id=JHIJNYORGPFFUCLRPFD3WR2LYT43ETZAVIOI5GVM4GPCI45JKN4A, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657110027 (190h13m20.088190158s) [noise=true , address=104.36.82.10:4079 , piece_id=X5BJHBJBPUYVTESEHTO4GDCSW2RL3IIPRAQYNHK3KSMVCN75PPHQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657115227 (190h13m20.088195358s) [noise=true , address=45.91.116.5:4297 , piece_id=WKO7WTAHEQM3QLNF5TPXEADX73HNP7HDQG3PXHNCIMJAODRT6HYA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657401574 (190h13m20.088481705s) [noise=true , address=45.91.116.12:4225 , piece_id=LYJANRVYRY3I47OFGYEE23U45M6SFABJ25VGPDYT237UX2G7OGEQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657454726 (190h13m20.088534857s) [noise=true , address=45.91.116.8:4047 , piece_id=RJC7VBLWQG3IWDT26WBHTU5FBBSYOG53AH6ZO3TFUGYHYXALDDFA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.657817942 (190h13m20.088898063s) [noise=true , address=104.36.82.14:4221 , piece_id=FZFP6Y74VTJSUD2YEQF6FBLTI2CWFYNB2JF6VYZPRCCBN3D2EM3Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658124448 (190h13m20.089204579s) [noise=true , address=45.91.116.13:4153 , piece_id=ZWRR7D73LCDRFOQ3QLVNOUQ2KBUDEEQFGD5YQ5Y7Y75IYIOCYXKQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658413160 (190h13m20.089493291s) [noise=true , address=77.90.31.18:4805 , piece_id=UH7UO5HDGUEG7X6TUZB44CF42B7P5VBOAMKOYBHED6WUSA2RJGDA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658680882 (190h13m20.089761013s) [noise=true , address=104.36.82.15:4259 , piece_id=FWIBYGGTKQ6E4ABW2C7YYMBRCA67MSXGCKZYBSLPLRNUL7ES4CTQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.658942383 (190h13m20.090022514s) [noise=true , address=104.36.82.7:4909 , piece_id=PR6W2X3DVXB3P7WVI7SHQTBZX5X3MBFFHA5F332LGM2WT2CI3SKA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659148467 (190h13m20.090228598s) [noise=true , address=77.90.31.21:4877 , piece_id=34ACLNZFPGPOT7S7L3OVFKJVWMTALBDO22GDSMSB6TEK2HZRQR4Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659344572 (190h13m20.090424703s) [noise=true , address=104.36.82.15:4258 , piece_id=6H3A5SEBLV33R2QZA3QWSHEE4ABVWBQY2EEKPF6HRNSDTP4LBC7Q, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659551488 (190h13m20.090631619s) [noise=true , address=104.36.82.4:4801 , piece_id=B5N63JGP5CZQIH6SDKWX5LYUWGV5VQ6DZNS25VI7QFBZFZEV3BMQ, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.659782891 (190h13m20.090863012s) [noise=true , address=77.90.31.13:4626 , piece_id=IU2ID4RHQMDDTMUGXI73TJ6CVTIH6WN4PISAPKLJOIZXT44TRWRA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:25.662332639 (190h13m20.09341277s) [noise=true , address=104.36.82.11:4113 , piece_id=OGQYTORMGCPWWUA2LS3WI3IVETZLW3B6AT2D3OJT4SDS2YA5YTXA, seg_pos={0 0}, upload=197906]: Uploading piece... Aug 16 15:39:27.442171003 (190h13m21.873251145s) [upload=197906]: Next returned nil segment Aug 16 15:39:27.442208184 (190h13m21.873288335s) [upload=197906]: Waiting for error group managing segments... Aug 16 15:39:27.725845033 (190h13m22.156925154s) [noise=true , address=104.36.82.15:4259 , piece_id=FWIBYGGTKQ6E4ABW2C7YYMBRCA67MSXGCKZYBSLPLRNUL7ES4CTQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.859612854 (190h13m22.290692985s) [noise=true , address=104.36.82.7:4909 , piece_id=PR6W2X3DVXB3P7WVI7SHQTBZX5X3MBFFHA5F332LGM2WT2CI3SKA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.860933421 (190h13m22.292013542s) [noise=true , address=104.36.82.10:4079 , piece_id=X5BJHBJBPUYVTESEHTO4GDCSW2RL3IIPRAQYNHK3KSMVCN75PPHQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.876058220 (190h13m22.307138341s) [noise=true , address=104.36.82.11:4113 , piece_id=OGQYTORMGCPWWUA2LS3WI3IVETZLW3B6AT2D3OJT4SDS2YA5YTXA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:27.935779070 (190h13m22.366859191s) [noise=true , address=104.36.82.15:4258 , piece_id=6H3A5SEBLV33R2QZA3QWSHEE4ABVWBQY2EEKPF6HRNSDTP4LBC7Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:28.519964965 (190h13m22.951045096s) [noise=true , address=45.91.116.5:4297 , piece_id=WKO7WTAHEQM3QLNF5TPXEADX73HNP7HDQG3PXHNCIMJAODRT6HYA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:28.759695192 (190h13m23.190775323s) [noise=true , address=77.90.31.13:4626 , piece_id=IU2ID4RHQMDDTMUGXI73TJ6CVTIH6WN4PISAPKLJOIZXT44TRWRA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.019266463 (190h13m23.450346584s) [noise=true , address=45.91.116.8:4047 , piece_id=RJC7VBLWQG3IWDT26WBHTU5FBBSYOG53AH6ZO3TFUGYHYXALDDFA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.062913397 (190h13m23.493993528s) [noise=true , address=77.90.31.18:4805 , piece_id=UH7UO5HDGUEG7X6TUZB44CF42B7P5VBOAMKOYBHED6WUSA2RJGDA, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:29.076663566 (190h13m23.507743688s) [noise=true , address=45.91.116.12:4225 , piece_id=LYJANRVYRY3I47OFGYEE23U45M6SFABJ25VGPDYT237UX2G7OGEQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:31.421888487 (190h13m25.852968628s) [noise=true , address=45.91.116.13:4153 , piece_id=ZWRR7D73LCDRFOQ3QLVNOUQ2KBUDEEQFGD5YQ5Y7Y75IYIOCYXKQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:54.210295384 (190h13m48.641375515s) [noise=true , address=77.90.31.21:4877 , piece_id=34ACLNZFPGPOT7S7L3OVFKJVWMTALBDO22GDSMSB6TEK2HZRQR4Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: <nil> Aug 16 15:39:54.764506344 (190h13m49.195586475s) [noise=true , address=104.36.82.9:4007 , piece_id=JHIJNYORGPFFUCLRPFD3WR2LYT43ETZAVIOI5GVM4GPCI45JKN4A, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764576648 (190h13m49.195656779s) [noise=true , address=104.36.82.8:4969 , piece_id=VWTIRBFPGJYQ45QU5HDOQSJVE2RKRRHH7OEHVHTBNCBH2Q4V65AQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764601335 (190h13m49.195681466s) [noise=true , address=104.36.82.4:4801 , piece_id=B5N63JGP5CZQIH6SDKWX5LYUWGV5VQ6DZNS25VI7QFBZFZEV3BMQ, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:47940->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764659647 (190h13m49.195739768s) [noise=true , address=104.36.82.14:4221 , piece_id=FZFP6Y74VTJSUD2YEQF6FBLTI2CWFYNB2JF6VYZPRCCBN3D2EM3Q, seg_pos={0 0}, upload=197906]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:34480->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:54.764683572 (190h13m49.195763693s) [seg_pos={0 0}, upload=197906]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:39:55.006122002 (190h13m49.437202123s) [upload=197906]: Done uploading segments... Aug 16 15:39:55.006141679 (190h13m49.43722181s) [upload=197906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243632082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...........X......X...X.X......X..X......X..X...XXXXXXXXXXX] duration: 1m0.235040345s FAIL (2025-08-16 15:39:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:38:44.300483419 (190h12m38.73156354s) [upload=197894]: Starting upload Aug 16 15:38:44.300526882 (190h12m38.731607013s) [upload=197894]: Done starting upload Aug 16 15:38:44.300656280 (190h12m38.731736411s) [upload=197894]: Uploading segments... Aug 16 15:38:44.300795907 (190h12m38.731876038s) [upload=197894]: Got next segment. Inline: false Aug 16 15:38:44.300861352 (190h12m38.731941483s) [upload=197894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:38:44.300866522 (190h12m38.731946643s) [upload=197894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:38:44.743510072 (190h12m39.174590203s) [seg_pos={0 0}, upload=197894]: Begin upload segment... Aug 16 15:38:44.743548054 (190h12m39.174628225s) [seg_pos={0 0}, upload=197894]: Done begin upload segment. Aug 16 15:38:44.743567632 (190h12m39.174647753s) [noise=true , address=77.90.31.15:4720 , piece_id=32V6EWBEYZBC5XMBTGYUICI4XCT6LDSUMDSX2CS3JWG4ZNPXJPRQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.743574996 (190h12m39.174655127s) [noise=true , address=104.36.82.4:4828 , piece_id=YBHELKMAAQMYZUTMVA3T7HZ5EIEDUOPBB3TZTTZWZP3ZOYY2OJWQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.743618769 (190h12m39.1746989s) [noise=true , address=77.90.31.19:4852 , piece_id=7JARS3DAZAAWPZG47XAI42SVQ3CFULKIL3EF6TW7YQ566P4BLYZQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.743900559 (190h12m39.17498068s) [noise=true , address=79.127.201.215:4357 , piece_id=H76O75YHIGIYRYH3QGJMPVDCZEVN33ZH2BCYGBQECJRW66DNG7FQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.743949472 (190h12m39.175029604s) [noise=true , address=104.36.82.59:4302 , piece_id=QEDIFSBPBCYCF3XQF2LFKEUXLE4YOIQW3BAV3CORTPZIYW5EIKCQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744156157 (190h12m39.175236288s) [noise=true , address=104.36.82.34:4573 , piece_id=47RTYGNXXGW22YX3RP52MURE7NDEIXREGAW2FWUPWEGDCZK4KKTQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744244626 (190h12m39.175324758s) [noise=true , address=104.36.82.58:4295 , piece_id=LV7LN7EJUGKDS7M64GF7XRPZTX4344LSDXCPMCX4QYFUFJHDTJGQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744452204 (190h12m39.175532325s) [noise=true , address=45.91.116.14:4204 , piece_id=ICDILLFEVSWWSKQURAFAVWEM6NKETLH3IBWA54QM46CIJKKU45IQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744585028 (190h12m39.175665159s) [noise=true , address=45.91.116.12:4248 , piece_id=U2B4MPVNEGBXDVZ2Y2F3DSW3R7N2AGYV4UGWLJ7LTNEUUYJLUITQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744773970 (190h12m39.175854101s) [noise=true , address=104.36.82.60:4342 , piece_id=WNIWGE7GJN6GACBZTVCPEKYBRCNW2EGPTUSAW2FJCBB6FK2CZOVA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.744885774 (190h12m39.175965905s) [noise=true , address=104.36.82.44:4888 , piece_id=D2XW5X63UKY7JZKPHDKSHAEHTXF6YLS7OKBTKBHX3AIJEIERNNWQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745018849 (190h12m39.17609898s) [noise=true , address=66.23.193.175:4561 , piece_id=NLIL4O6ZQASACUT3N3SSRW4NBVKS6GVPH64MBHQKCI2XID2F4LNQ, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745228800 (190h12m39.176308921s) [noise=true , address=77.90.31.11:4567 , piece_id=OGMG6HVZSQL3Z3TG55TSMVFIPEPXMZWTZI7W6U7TJPFF7PUDBZXA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745232366 (190h12m39.176312497s) [noise=true , address=45.91.116.9:4294 , piece_id=YWRRVHW22KNZ6SIMFE5FBEIQOUM5YE3Q6FHFVQCFAWMAQ5K5M4JA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745442449 (190h12m39.17652258s) [noise=true , address=77.90.31.13:4632 , piece_id=CIW5J6FM3CIX7O35VT65VEXORWPUVDA4RM2BFJXPW2TRRZDE3ESA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745532631 (190h12m39.176612762s) [noise=true , address=104.36.82.3:4793 , piece_id=FLJO6TYVKOZTNC57GOXQ6MTARTIJGHMUSG56ZGQRQFCMXIRLMP3A, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745701454 (190h12m39.176781585s) [noise=true , address=104.36.82.23:4451 , piece_id=2X74YBS7T5L2GKK7KFL6CC67C7LFIJQZ5ZYF3NOUMOSQNSE273FA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:44.745954168 (190h12m39.177034299s) [noise=true , address=77.90.31.21:4905 , piece_id=7UPCJRL34TMVCADZVCEKXP2H2NEI7IG5R6AYXZDR2H2WNSHMOJZA, seg_pos={0 0}, upload=197894]: Uploading piece... Aug 16 15:38:45.941918996 (190h12m40.372999127s) [upload=197894]: Next returned nil segment Aug 16 15:38:45.941939906 (190h12m40.373020027s) [upload=197894]: Waiting for error group managing segments... Aug 16 15:38:46.670903238 (190h12m41.101983359s) [noise=true , address=77.90.31.19:4852 , piece_id=7JARS3DAZAAWPZG47XAI42SVQ3CFULKIL3EF6TW7YQ566P4BLYZQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:46.823377220 (190h12m41.254457401s) [noise=true , address=77.90.31.13:4632 , piece_id=CIW5J6FM3CIX7O35VT65VEXORWPUVDA4RM2BFJXPW2TRRZDE3ESA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:47.254982890 (190h12m41.686063021s) [noise=true , address=77.90.31.11:4567 , piece_id=OGMG6HVZSQL3Z3TG55TSMVFIPEPXMZWTZI7W6U7TJPFF7PUDBZXA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:50.122450636 (190h12m44.553530767s) [noise=true , address=77.90.31.21:4905 , piece_id=7UPCJRL34TMVCADZVCEKXP2H2NEI7IG5R6AYXZDR2H2WNSHMOJZA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:50.895540830 (190h12m45.326620961s) [noise=true , address=45.91.116.14:4204 , piece_id=ICDILLFEVSWWSKQURAFAVWEM6NKETLH3IBWA54QM46CIJKKU45IQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:51.560072167 (190h12m45.991152299s) [noise=true , address=45.91.116.9:4294 , piece_id=YWRRVHW22KNZ6SIMFE5FBEIQOUM5YE3Q6FHFVQCFAWMAQ5K5M4JA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:51.597990530 (190h12m46.029070661s) [noise=true , address=45.91.116.12:4248 , piece_id=U2B4MPVNEGBXDVZ2Y2F3DSW3R7N2AGYV4UGWLJ7LTNEUUYJLUITQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:52.569897627 (190h12m47.000977758s) [noise=true , address=79.127.201.215:4357 , piece_id=H76O75YHIGIYRYH3QGJMPVDCZEVN33ZH2BCYGBQECJRW66DNG7FQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:38:53.994271149 (190h12m48.425351321s) [noise=true , address=66.23.193.175:4561 , piece_id=NLIL4O6ZQASACUT3N3SSRW4NBVKS6GVPH64MBHQKCI2XID2F4LNQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: <nil> Aug 16 15:39:43.658480556 (190h13m38.089560687s) [noise=true , address=104.36.82.34:4573 , piece_id=47RTYGNXXGW22YX3RP52MURE7NDEIXREGAW2FWUPWEGDCZK4KKTQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:1yD3c1tEtGt2yYEyzDX5GugUChtkCQa9VTWkBH9ftgwvqn72SU, address:104.36.82.34:4573): protocol: rpc: dial tcp 104.36.82.34:4573: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658532506 (190h13m38.089612637s) [noise=true , address=104.36.82.4:4828 , piece_id=YBHELKMAAQMYZUTMVA3T7HZ5EIEDUOPBB3TZTTZWZP3ZOYY2OJWQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:44112->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658612108 (190h13m38.089692229s) [noise=true , address=104.36.82.44:4888 , piece_id=D2XW5X63UKY7JZKPHDKSHAEHTXF6YLS7OKBTKBHX3AIJEIERNNWQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: write tcp 188.166.225.170:50068->104.36.82.44:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658666232 (190h13m38.089746363s) [noise=true , address=104.36.82.3:4793 , piece_id=FLJO6TYVKOZTNC57GOXQ6MTARTIJGHMUSG56ZGQRQFCMXIRLMP3A, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:104.36.82.3:4793): protocol: write tcp 188.166.225.170:43328->104.36.82.3:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658701649 (190h13m38.08978178s) [noise=true , address=104.36.82.60:4342 , piece_id=WNIWGE7GJN6GACBZTVCPEKYBRCNW2EGPTUSAW2FJCBB6FK2CZOVA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:104.36.82.60:4342): protocol: write tcp 188.166.225.170:58838->104.36.82.60:4342: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658705476 (190h13m38.089785607s) [noise=true , address=77.90.31.15:4720 , piece_id=32V6EWBEYZBC5XMBTGYUICI4XCT6LDSUMDSX2CS3JWG4ZNPXJPRQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 188.166.225.170:33560->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658742307 (190h13m38.089822438s) [noise=true , address=104.36.82.59:4302 , piece_id=QEDIFSBPBCYCF3XQF2LFKEUXLE4YOIQW3BAV3CORTPZIYW5EIKCQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:104.36.82.59:4302): protocol: write tcp 188.166.225.170:52152->104.36.82.59:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658762235 (190h13m38.089842356s) [noise=true , address=104.36.82.58:4295 , piece_id=LV7LN7EJUGKDS7M64GF7XRPZTX4344LSDXCPMCX4QYFUFJHDTJGQ, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:1oLa34qwPBAx3CTvYYj7GN3oxxWGJ9dAYXfke1eUftZc6xzfqg, address:104.36.82.58:4295): protocol: write tcp 188.166.225.170:54494->104.36.82.58:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:43.658806830 (190h13m38.089886961s) [noise=true , address=104.36.82.23:4451 , piece_id=2X74YBS7T5L2GKK7KFL6CC67C7LFIJQZ5ZYF3NOUMOSQNSE273FA, seg_pos={0 0}, upload=197894]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:39:43.658824014 (190h13m38.089904145s) [seg_pos={0 0}, upload=197894]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:39:43.892440874 (190h13m38.323521005s) [upload=197894]: Done uploading segments... Aug 16 15:39:43.892465752 (190h13m38.323545873s) [upload=197894]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.235050785s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.296108265s FAIL (2025-08-16 15:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.762218403 (190h12m30.193298534s) [noise=true , address=77.90.31.11:4557 , piece_id=ZDEUS3DPITNK2CQGN74QT6PMGWZJ4LXLX32GOW26NJM3NL56NGUA, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.762594232 (190h12m30.193674363s) [noise=true , address=104.36.82.5:4869 , piece_id=PSEMVOPWF7TKSHLRDI6CUAZDKIY32NDAVET4QDI5ZUP5JJYXCKDQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.762598480 (190h12m30.193678611s) [noise=true , address=79.127.201.215:4353 , piece_id=K5N4TLLWD76DG4SIOFB6FD3GCUFXF77QFHNTBVQUS72I4F5NXOTQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.762869038 (190h12m30.193949159s) [noise=true , address=66.23.193.175:4556 , piece_id=RUEU4ZQCT4COQYD4VCLWTYGBOHZG43SN7APVB5LJ7HE5K7UTWURQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.763258773 (190h12m30.194338904s) [noise=true , address=79.127.220.101:4070 , piece_id=6XPBDQAB7LMRQFLX5Z6J7LFWRW3PILBA6BWC25QSDHSOXCA4KZOA, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.763532617 (190h12m30.194612748s) [noise=true , address=104.36.82.44:4887 , piece_id=LI4ETUJCXW6ZOXH4IENSSWNZNPL2LZVKDWQPPLSGJ3T3FEFJ5G4Q, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.763826009 (190h12m30.19490614s) [noise=true , address=45.91.116.5:4324 , piece_id=MWFMHGR6ZAAERAIU7EZIK4ZWS6BBLPRH5SUD6BIZMEWMIX6TIBGA, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.764184355 (190h12m30.195264476s) [noise=true , address=104.36.82.23:4450 , piece_id=JOKG3XGDEMZNUGGFQT5HLXNBNI7ULSZ2LWD4DGGEYZCULUBJMZEQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.764454502 (190h12m30.195534633s) [noise=true , address=77.90.31.15:4727 , piece_id=EKNM2WIGLSWG5IC3VYWGSHY5TROM7CUGXEK4USCD5M37HEUERWZQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.764694931 (190h12m30.195775062s) [noise=true , address=104.36.82.35:4601 , piece_id=3O4OKHHDZKKBVCKIIYXXKCGJBJRIL747AKHQODHRVOE22HWSE37Q, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.764902328 (190h12m30.195982449s) [noise=true , address=104.36.82.36:4649 , piece_id=JCNIY5MTFX7YFST6PV7QXA4EE6EZO47OVWEGKMCH4WEKKNL5YTNA, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.765088695 (190h12m30.196168826s) [noise=true , address=79.127.220.101:4076 , piece_id=T5QGE6I4CPS3JHJQBREN4NB5QIM26PBI2C2LG7UEMSMK4EUUFOWQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.765279190 (190h12m30.196359321s) [noise=true , address=45.91.116.9:4294 , piece_id=PGQM2S3IHIKH6VZYIW2UB4O7OTK6PV6C62JNRHUO3F2BKPIS3EMQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.765463381 (190h12m30.196543512s) [noise=true , address=104.36.82.39:4725 , piece_id=YVYAQ4Y224UOHWO7PUDPETR7PM4QFWNDSNOHUNMKU5TLZVPUJ4FQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.765643767 (190h12m30.196723898s) [noise=true , address=77.90.31.13:4659 , piece_id=WGLVBIMV5FIRQWGSZRRMD5NTNT3OWWJKCMMMOAN5DYHPDNPR374Q, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.765846665 (190h12m30.196926796s) [noise=true , address=77.90.31.16:4757 , piece_id=7FXVEC2FBKHDP4PJP5VXLR6YQ54HBG6BRPEVPBAMGYNDYNELI2BA, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:35.766036268 (190h12m30.197116399s) [noise=true , address=104.36.82.60:4363 , piece_id=XZWRRB5XN2OHVI6FEJQOW43QBELR3ZVJEHXGK424BQBY4DL4XXGQ, seg_pos={0 0}, upload=197891]: Uploading piece... Aug 16 15:38:36.928685650 (190h12m31.359765801s) [upload=197891]: Next returned nil segment Aug 16 15:38:36.928704046 (190h12m31.359784217s) [upload=197891]: Waiting for error group managing segments... Aug 16 15:38:37.952617603 (190h12m32.383697754s) [noise=true , address=77.90.31.11:4557 , piece_id=ZDEUS3DPITNK2CQGN74QT6PMGWZJ4LXLX32GOW26NJM3NL56NGUA, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:38.036503950 (190h12m32.467584081s) [noise=true , address=77.90.31.13:4659 , piece_id=WGLVBIMV5FIRQWGSZRRMD5NTNT3OWWJKCMMMOAN5DYHPDNPR374Q, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:38.183203453 (190h12m32.614283584s) [noise=true , address=77.90.31.16:4757 , piece_id=7FXVEC2FBKHDP4PJP5VXLR6YQ54HBG6BRPEVPBAMGYNDYNELI2BA, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:38.790329683 (190h12m33.221409814s) [noise=true , address=45.91.116.5:4324 , piece_id=MWFMHGR6ZAAERAIU7EZIK4ZWS6BBLPRH5SUD6BIZMEWMIX6TIBGA, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:39.473677465 (190h12m33.904757596s) [noise=true , address=79.127.220.101:4070 , piece_id=6XPBDQAB7LMRQFLX5Z6J7LFWRW3PILBA6BWC25QSDHSOXCA4KZOA, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:39.906899356 (190h12m34.337979487s) [noise=true , address=79.127.220.101:4076 , piece_id=T5QGE6I4CPS3JHJQBREN4NB5QIM26PBI2C2LG7UEMSMK4EUUFOWQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:43.062257308 (190h12m37.493337439s) [noise=true , address=79.127.201.215:4353 , piece_id=K5N4TLLWD76DG4SIOFB6FD3GCUFXF77QFHNTBVQUS72I4F5NXOTQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:44.330131176 (190h12m38.761211307s) [noise=true , address=45.91.116.9:4294 , piece_id=PGQM2S3IHIKH6VZYIW2UB4O7OTK6PV6C62JNRHUO3F2BKPIS3EMQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:52.289905473 (190h12m46.720985614s) [noise=true , address=66.23.193.175:4556 , piece_id=RUEU4ZQCT4COQYD4VCLWTYGBOHZG43SN7APVB5LJ7HE5K7UTWURQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:38:56.642777838 (190h12m51.073857969s) [noise=true , address=104.36.82.5:4869 , piece_id=PSEMVOPWF7TKSHLRDI6CUAZDKIY32NDAVET4QDI5ZUP5JJYXCKDQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: <nil> Aug 16 15:39:34.626386293 (190h13m29.057466424s) [noise=true , address=104.36.82.60:4363 , piece_id=XZWRRB5XN2OHVI6FEJQOW43QBELR3ZVJEHXGK424BQBY4DL4XXGQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: rpc: dial tcp 104.36.82.60:4363: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626427281 (190h13m29.057507412s) [noise=true , address=104.36.82.39:4725 , piece_id=YVYAQ4Y224UOHWO7PUDPETR7PM4QFWNDSNOHUNMKU5TLZVPUJ4FQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:104.36.82.39:4725): protocol: rpc: dial tcp 104.36.82.39:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626463310 (190h13m29.057543442s) [noise=true , address=104.36.82.58:4293 , piece_id=TMUMQ6GKDYSR53DW2WPFKYNBHOKUFS26JDSLBK3G6W4TMVFI2YQQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: rpc: dial tcp 104.36.82.58:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626488518 (190h13m29.057568649s) [noise=true , address=77.90.31.15:4727 , piece_id=EKNM2WIGLSWG5IC3VYWGSHY5TROM7CUGXEK4USCD5M37HEUERWZQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 188.166.225.170:40112->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626496093 (190h13m29.057576224s) [noise=true , address=104.36.82.44:4887 , piece_id=LI4ETUJCXW6ZOXH4IENSSWNZNPL2LZVKDWQPPLSGJ3T3FEFJ5G4Q, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 188.166.225.170:56300->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626532382 (190h13m29.057612513s) [noise=true , address=104.36.82.36:4649 , piece_id=JCNIY5MTFX7YFST6PV7QXA4EE6EZO47OVWEGKMCH4WEKKNL5YTNA, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: rpc: dial tcp 104.36.82.36:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626550056 (190h13m29.057630187s) [noise=true , address=104.36.82.35:4601 , piece_id=3O4OKHHDZKKBVCKIIYXXKCGJBJRIL747AKHQODHRVOE22HWSE37Q, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:16ggANPGWDLtyxp9hMkCsNYueB2KuVWsKVrG8mFa6pPBXW5RiW, address:104.36.82.35:4601): protocol: rpc: dial tcp 104.36.82.35:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626575725 (190h13m29.057655917s) [noise=true , address=104.36.82.23:4450 , piece_id=JOKG3XGDEMZNUGGFQT5HLXNBNI7ULSZ2LWD4DGGEYZCULUBJMZEQ, seg_pos={0 0}, upload=197891]: Done uploading piece. err: ecclient: upload failed (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms, address:104.36.82.23:4450): protocol: rpc: dial tcp 104.36.82.23:4450: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:39:34.626611283 (190h13m29.057691414s) [seg_pos={0 0}, upload=197891]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:39:34.920526069 (190h13m29.3516062s) [upload=197891]: Done uploading segments... Aug 16 15:39:34.920549394 (190h13m29.351629525s) [upload=197891]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.296121561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [......X...X...X..................X.X..X.....X...............] duration: 6.547466823s FAIL (2025-08-16 15:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 6.547493284s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........................................X......X..XXXXXXXXX] duration: 30.236005439s FAIL (2025-08-16 15:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:38:27.143153725 (190h12m21.574233856s) [upload=197887]: Starting upload Aug 16 15:38:27.143210113 (190h12m21.574290244s) [upload=197887]: Done starting upload Aug 16 15:38:27.143288683 (190h12m21.574368804s) [upload=197887]: Uploading segments... Aug 16 15:38:27.143434642 (190h12m21.574514773s) [upload=197887]: Got next segment. Inline: false Aug 16 15:38:27.143460311 (190h12m21.574540442s) [upload=197887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:38:27.143464479 (190h12m21.57454461s) [upload=197887]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:38:27.380097531 (190h12m21.811177652s) [seg_pos={0 0}, upload=197887]: Begin upload segment... Aug 16 15:38:27.380142506 (190h12m21.811222637s) [seg_pos={0 0}, upload=197887]: Done begin upload segment. Aug 16 15:38:27.380162755 (190h12m21.811242886s) [noise=true , address=45.91.116.4:4939 , piece_id=U237ORXS3VQBY3YLEN6S53LSAN3PLFMCZOYU6WROGPXK7ZAWSGTQ, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.380173125 (190h12m21.811253256s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQGXVDFOQLZJRR4JFXSACBRNIYHLUAOXOC5RH3UQK7LTC7GCLYA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.380577659 (190h12m21.81165779s) [noise=true , address=104.36.82.8:4972 , piece_id=XRQARB7VU5XYHF266UYDAQHUBPQ5MYMOMJ2BFJHJFALWIHNWENPA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.380582719 (190h12m21.81166285s) [noise=true , address=104.36.82.14:4221 , piece_id=3YRTCZPHF642P24OWLWESICIRMSYFBDP7QYHADXSHDNHCRJWCCSA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.380864629 (190h12m21.81194476s) [noise=true , address=45.91.116.9:4263 , piece_id=NHWB47EW523P2XISAK2UMGBWBK2PIIZEJVFPHSAVGJ4WCZW72UGA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.380900237 (190h12m21.811980368s) [noise=true , address=104.36.82.15:4257 , piece_id=EDF7MKF4BALRDYF2Y3UFHIXJNY2Y6UDKXRB5L76OJMWPPG3AXVWQ, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381153200 (190h12m21.812233321s) [noise=true , address=45.91.116.13:4154 , piece_id=H4USQI42FCWQAOY4KVZM4BFEZAHS73H3NYLQUTYZPNPN5XLX6PRA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381165985 (190h12m21.812246116s) [noise=true , address=45.91.116.12:4226 , piece_id=IZW6NTVSHHERILRCAJPIAKKAEQL4OGQJA4ML62KK7QSJEWLM7NWQ, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381375636 (190h12m21.812455767s) [noise=true , address=104.36.82.2:4731 , piece_id=FMFPTZLFRGICHMWXTFFFKRACAMBBJMM2LA2VLA7GLNORVRTQYG6A, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381393200 (190h12m21.812473331s) [noise=true , address=77.90.31.15:4697 , piece_id=D5O7HPNH6CTCBKYX57T7ZFOYS7DCWERMEBX22YHNLZBP2D5DNQXQ, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381584476 (190h12m21.812664607s) [noise=true , address=104.36.82.11:4113 , piece_id=KZ6EY7RNYRIKRGPJQDP34JFJLIZDE3AF5HWUVYOOXPZF6OGVVK6Q, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381615465 (190h12m21.812695596s) [noise=true , address=104.36.82.7:4909 , piece_id=IJMCCJHW3VB2JUPGVBHICHAR32E2ES3EOL3D6NRI3BTGDRXZK6KA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.381876705 (190h12m21.812956826s) [noise=true , address=104.36.82.6:4873 , piece_id=4AWQZJ2UULCNPJOQ6DSSIRDRUQMAJ6JHP456YJMP6SFOBEFTMQTA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.383443523 (190h12m21.814523654s) [noise=true , address=104.36.82.12:4149 , piece_id=HIWEXFEBYQJRWWNYAC2BF2OFBCHDNP2WUSM3RG2V7R44AJQAFQLA, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.384037769 (190h12m21.815117901s) [noise=true , address=104.36.82.9:4007 , piece_id=EJ5IKNN2VEUEASXTDK2BPBGTPWBSF4QCZE7IAMJYAEEXVC5YM3PQ, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:27.384445119 (190h12m21.81552525s) [noise=true , address=104.36.82.4:4801 , piece_id=O6TAVENTKW575NNLZDTK5PH72KBJNS4QVAEFAAHLJHB4NUL43U4Q, seg_pos={0 0}, upload=197887]: Uploading piece... Aug 16 15:38:29.027459397 (190h12m23.458539528s) [upload=197887]: Next returned nil segment Aug 16 15:38:29.027475207 (190h12m23.458555328s) [upload=197887]: Waiting for error group managing segments... Aug 16 15:38:29.429780845 (190h12m23.860860966s) [noise=true , address=104.36.82.15:4257 , piece_id=EDF7MKF4BALRDYF2Y3UFHIXJNY2Y6UDKXRB5L76OJMWPPG3AXVWQ, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.445542604 (190h12m23.876622735s) [noise=true , address=104.36.82.7:4909 , piece_id=IJMCCJHW3VB2JUPGVBHICHAR32E2ES3EOL3D6NRI3BTGDRXZK6KA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.494206577 (190h12m23.925286708s) [noise=true , address=104.36.82.12:4149 , piece_id=HIWEXFEBYQJRWWNYAC2BF2OFBCHDNP2WUSM3RG2V7R44AJQAFQLA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.517829076 (190h12m23.948909207s) [noise=true , address=104.36.82.6:4873 , piece_id=4AWQZJ2UULCNPJOQ6DSSIRDRUQMAJ6JHP456YJMP6SFOBEFTMQTA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.564793407 (190h12m23.995873528s) [noise=true , address=104.36.82.11:4113 , piece_id=KZ6EY7RNYRIKRGPJQDP34JFJLIZDE3AF5HWUVYOOXPZF6OGVVK6Q, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.640361749 (190h12m24.07144188s) [noise=true , address=45.91.116.4:4939 , piece_id=U237ORXS3VQBY3YLEN6S53LSAN3PLFMCZOYU6WROGPXK7ZAWSGTQ, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.740103263 (190h12m24.171183394s) [noise=true , address=45.91.116.12:4226 , piece_id=IZW6NTVSHHERILRCAJPIAKKAEQL4OGQJA4ML62KK7QSJEWLM7NWQ, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:29.959083127 (190h12m24.390163248s) [noise=true , address=45.91.116.9:4263 , piece_id=NHWB47EW523P2XISAK2UMGBWBK2PIIZEJVFPHSAVGJ4WCZW72UGA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:30.429581013 (190h12m24.860661144s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQGXVDFOQLZJRR4JFXSACBRNIYHLUAOXOC5RH3UQK7LTC7GCLYA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:32.002974310 (190h12m26.434054431s) [noise=true , address=45.91.116.13:4154 , piece_id=H4USQI42FCWQAOY4KVZM4BFEZAHS73H3NYLQUTYZPNPN5XLX6PRA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: <nil> Aug 16 15:38:56.487791899 (190h12m50.91887203s) [noise=true , address=104.36.82.8:4972 , piece_id=XRQARB7VU5XYHF266UYDAQHUBPQ5MYMOMJ2BFJHJFALWIHNWENPA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.487844219 (190h12m50.91892435s) [noise=true , address=104.36.82.9:4007 , piece_id=EJ5IKNN2VEUEASXTDK2BPBGTPWBSF4QCZE7IAMJYAEEXVC5YM3PQ, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.487882223 (190h12m50.918962394s) [noise=true , address=104.36.82.14:4221 , piece_id=3YRTCZPHF642P24OWLWESICIRMSYFBDP7QYHADXSHDNHCRJWCCSA, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:54284->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.487898994 (190h12m50.918979125s) [noise=true , address=77.90.31.15:4697 , piece_id=D5O7HPNH6CTCBKYX57T7ZFOYS7DCWERMEBX22YHNLZBP2D5DNQXQ, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49394->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.487970341 (190h12m50.919050472s) [noise=true , address=104.36.82.4:4801 , piece_id=O6TAVENTKW575NNLZDTK5PH72KBJNS4QVAEFAAHLJHB4NUL43U4Q, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:34584->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.487975731 (190h12m50.919055862s) [noise=true , address=104.36.82.2:4731 , piece_id=FMFPTZLFRGICHMWXTFFFKRACAMBBJMM2LA2VLA7GLNORVRTQYG6A, seg_pos={0 0}, upload=197887]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:56.488001671 (190h12m50.919081832s) [seg_pos={0 0}, upload=197887]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:38:56.721770522 (190h12m51.152850653s) [upload=197887]: Done uploading segments... Aug 16 15:38:56.721792194 (190h12m51.152872325s) [upload=197887]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.236025146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...........X......X...X.X......X..X......X..X...XXXXXXXXXX] duration: 1m0.412562082s FAIL (2025-08-16 15:38:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:37:43.637488681 (190h11m38.068568812s) [upload=197876]: Starting upload Aug 16 15:37:43.637537314 (190h11m38.068617445s) [upload=197876]: Done starting upload Aug 16 15:37:43.637672082 (190h11m38.068752213s) [upload=197876]: Uploading segments... Aug 16 15:37:43.637814715 (190h11m38.068894846s) [upload=197876]: Got next segment. Inline: false Aug 16 15:37:43.637849813 (190h11m38.068929944s) [upload=197876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:37:43.637854191 (190h11m38.068934322s) [upload=197876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:37:44.502652970 (190h11m38.933733101s) [seg_pos={0 0}, upload=197876]: Begin upload segment... Aug 16 15:37:44.502700680 (190h11m38.933780811s) [seg_pos={0 0}, upload=197876]: Done begin upload segment. Aug 16 15:37:44.502727481 (190h11m38.933807612s) [noise=true , address=104.36.82.5:4856 , piece_id=3A5B5JPP5D7T4FUFM47N32THAKM63SK75CCOFDPKRSR2AYXQMO3A, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.502738804 (190h11m38.933818935s) [noise=true , address=104.36.82.60:4357 , piece_id=DWZLLL3FGDECLRNQHRCHV5IR3MB3ILUGY2YEXXXR4CDZVAKTH5AA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503101187 (190h11m38.934181318s) [noise=true , address=104.36.82.58:4265 , piece_id=RUBAFJWANDZCS76YUNKJSI7NNXGZPAKQYJORMKVWZ752TCV3RCRA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503374179 (190h11m38.93445431s) [noise=true , address=104.36.82.4:4807 , piece_id=ACNO6UR7EGZSQVTYM4QOQ4BGX4AKIUZ6YC355DFWZIK3PERF6VCA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503402844 (190h11m38.934482995s) [noise=true , address=104.36.82.23:4474 , piece_id=APUB2X7TPYG6YERH7RKA2IMDG3XCKKTIU6AWB4OBJ4LJJFYBL6IQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503639227 (190h11m38.934719368s) [noise=true , address=79.127.201.215:4345 , piece_id=X5C2F5O2SGQFXH4FXE6QNQQ65L4GGVHYCRZJ37K46P7IFEGLDLJQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503881680 (190h11m38.934961811s) [noise=true , address=77.90.31.10:4534 , piece_id=CE7P7QCQ5ZFFB5LZ2VIEXR24WXOANWGVGDGCKMBJGUMOK5UHLXHQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.503937897 (190h11m38.935018028s) [noise=true , address=104.36.82.44:4875 , piece_id=VKOKNL3MWC3THVUAJVBG64PAORUR5T2UWW3JX5KNUMXU6F5D36MQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.504110618 (190h11m38.93519075s) [noise=true , address=66.23.193.175:4553 , piece_id=PRCFBS2YSFMSCWTKUZB4UQ3QTTLTKJVDDNTTCQRNDHWCDNIPWV3Q, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.504390544 (190h11m38.935470675s) [noise=true , address=104.36.82.59:4307 , piece_id=ZGR7MLHSMELVVLZSXGJARKOHEON2QUQFKBGPFGDJ7Z2GLJYZGUOQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.504526514 (190h11m38.935606655s) [noise=true , address=77.90.31.13:4637 , piece_id=Z4GTCUSKSK47JW5JRBWUFFRGJZWBUELIVZ7OHYGR7T6U73AZGBOQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.504632507 (190h11m38.935712628s) [noise=true , address=79.127.201.214:4320 , piece_id=RAVIIGYN4RBHVKDVWQEALYY6TMQQMKWWF2KQCR7O2WG7XQRROQTA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.504825316 (190h11m38.935905447s) [noise=true , address=77.90.31.12:4595 , piece_id=ZY2PTN4ZJJMABR2QOEP4OQSKPCEZWS3H6UL6PE6DBMPA23L53HTQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.505007143 (190h11m38.936087264s) [noise=true , address=104.36.82.22:4544 , piece_id=WLCK7RFEURJ37VHN7ZVVCQRML55YWKGTL5NXHQKEOCCVES7JRYQQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.505443358 (190h11m38.936523489s) [noise=true , address=77.90.31.19:4852 , piece_id=VEP3LFZLLIAFTWUEAJGGGPG6FEO3R4SBZLGT65BTHZTTPRPXSSGA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.505747560 (190h11m38.936827691s) [noise=true , address=104.36.82.9:4025 , piece_id=L55CDZWMCFQU4BYZS4OOVUMVECGQXHDP5TPRMISQZYSSPL4GEBWQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.505848393 (190h11m38.936928524s) [noise=true , address=104.36.82.36:4635 , piece_id=7HTU7DLJ3543SHHNTGKL7AZJEWV23KTNLZCOW27WESMCC7NVIPYQ, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:44.506239782 (190h11m38.937319913s) [noise=true , address=77.90.31.21:4881 , piece_id=D5AU2F67CEZHMPCGOIELIWTDCBGH4LLJVQQBBVXNN55IISGR36VA, seg_pos={0 0}, upload=197876]: Uploading piece... Aug 16 15:37:45.580487216 (190h11m40.011567347s) [noise=true , address=79.127.201.214:4320 , piece_id=RAVIIGYN4RBHVKDVWQEALYY6TMQQMKWWF2KQCR7O2WG7XQRROQTA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:12ZLCXYP6sppXZcmj49C8jjgXbLGzgyowtpb1um9XZguzfVnM2r, address:79.127.201.214:4320): protocol: storage node overloaded, request limit: 2000 Aug 16 15:37:45.659510408 (190h11m40.090590539s) [upload=197876]: Next returned nil segment Aug 16 15:37:45.659526149 (190h11m40.09060628s) [upload=197876]: Waiting for error group managing segments... Aug 16 15:37:46.665053041 (190h11m41.096133162s) [noise=true , address=77.90.31.12:4595 , piece_id=ZY2PTN4ZJJMABR2QOEP4OQSKPCEZWS3H6UL6PE6DBMPA23L53HTQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:37:46.689020010 (190h11m41.120100141s) [noise=true , address=77.90.31.13:4637 , piece_id=Z4GTCUSKSK47JW5JRBWUFFRGJZWBUELIVZ7OHYGR7T6U73AZGBOQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:37:46.819220676 (190h11m41.250300807s) [noise=true , address=77.90.31.19:4852 , piece_id=VEP3LFZLLIAFTWUEAJGGGPG6FEO3R4SBZLGT65BTHZTTPRPXSSGA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:37:46.898114200 (190h11m41.329194321s) [noise=true , address=77.90.31.21:4881 , piece_id=D5AU2F67CEZHMPCGOIELIWTDCBGH4LLJVQQBBVXNN55IISGR36VA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:37:49.028927377 (190h11m43.460007498s) [noise=true , address=79.127.201.215:4345 , piece_id=X5C2F5O2SGQFXH4FXE6QNQQ65L4GGVHYCRZJ37K46P7IFEGLDLJQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:37:53.236623122 (190h11m47.667703253s) [noise=true , address=66.23.193.175:4553 , piece_id=PRCFBS2YSFMSCWTKUZB4UQ3QTTLTKJVDDNTTCQRNDHWCDNIPWV3Q, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:38:04.174716664 (190h11m58.605796795s) [noise=true , address=77.90.31.10:4534 , piece_id=CE7P7QCQ5ZFFB5LZ2VIEXR24WXOANWGVGDGCKMBJGUMOK5UHLXHQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: <nil> Aug 16 15:38:42.981074834 (190h12m37.412154955s) [noise=true , address=104.36.82.23:4474 , piece_id=APUB2X7TPYG6YERH7RKA2IMDG3XCKKTIU6AWB4OBJ4LJJFYBL6IQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:41040->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981115232 (190h12m37.412195363s) [noise=true , address=104.36.82.44:4875 , piece_id=VKOKNL3MWC3THVUAJVBG64PAORUR5T2UWW3JX5KNUMXU6F5D36MQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:104.36.82.44:4875): protocol: write tcp 188.166.225.170:59914->104.36.82.44:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981145730 (190h12m37.412225861s) [noise=true , address=104.36.82.59:4307 , piece_id=ZGR7MLHSMELVVLZSXGJARKOHEON2QUQFKBGPFGDJ7Z2GLJYZGUOQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:127F8oKF6GRv6Rdht4BCPt4G9UhVXeWVHKGJRP4WrrkcfDMmApT, address:104.36.82.59:4307): protocol: write tcp 188.166.225.170:51166->104.36.82.59:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981178954 (190h12m37.412259075s) [noise=true , address=104.36.82.9:4025 , piece_id=L55CDZWMCFQU4BYZS4OOVUMVECGQXHDP5TPRMISQZYSSPL4GEBWQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 188.166.225.170:44918->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981186979 (190h12m37.41226711s) [noise=true , address=104.36.82.36:4635 , piece_id=7HTU7DLJ3543SHHNTGKL7AZJEWV23KTNLZCOW27WESMCC7NVIPYQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:104.36.82.36:4635): protocol: rpc: dial tcp 104.36.82.36:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981208781 (190h12m37.412288902s) [noise=true , address=104.36.82.5:4856 , piece_id=3A5B5JPP5D7T4FUFM47N32THAKM63SK75CCOFDPKRSR2AYXQMO3A, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: write tcp 188.166.225.170:59126->104.36.82.5:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981220232 (190h12m37.412300363s) [noise=true , address=104.36.82.22:4544 , piece_id=WLCK7RFEURJ37VHN7ZVVCQRML55YWKGTL5NXHQKEOCCVES7JRYQQ, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:1YkVN8PYQuRCwyhAfuMPy9qZaA7ZqWqj6BTNjtUSVxNw6eq4Sx, address:104.36.82.22:4544): protocol: write tcp 188.166.225.170:53736->104.36.82.22:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981238638 (190h12m37.412318769s) [noise=true , address=104.36.82.4:4807 , piece_id=ACNO6UR7EGZSQVTYM4QOQ4BGX4AKIUZ6YC355DFWZIK3PERF6VCA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:104.36.82.4:4807): protocol: rpc: dial tcp 104.36.82.4:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981253076 (190h12m37.412333197s) [noise=true , address=104.36.82.60:4357 , piece_id=DWZLLL3FGDECLRNQHRCHV5IR3MB3ILUGY2YEXXXR4CDZVAKTH5AA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:17EmBobdmk16xm9girzx3asYwkhdCKKZCCPrBLwWTS4ujMdGmi, address:104.36.82.60:4357): protocol: write tcp 188.166.225.170:57790->104.36.82.60:4357: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981269977 (190h12m37.412350108s) [noise=true , address=104.36.82.58:4265 , piece_id=RUBAFJWANDZCS76YUNKJSI7NNXGZPAKQYJORMKVWZ752TCV3RCRA, seg_pos={0 0}, upload=197876]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:104.36.82.58:4265): protocol: write tcp 188.166.225.170:38240->104.36.82.58:4265: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:42.981289355 (190h12m37.412369486s) [seg_pos={0 0}, upload=197876]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:38:43.392052546 (190h12m37.823132667s) [upload=197876]: Done uploading segments... Aug 16 15:38:43.392080820 (190h12m37.823160951s) [upload=197876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.412584404s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.230808587s FAIL (2025-08-16 15:38:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ading piece... Aug 16 15:37:35.242964647 (190h11m29.674044788s) [noise=true , address=77.90.31.21:4882 , piece_id=TTG6ZYGQOJFR2P4K2NHOJ7DFDAZUFWCC7U64QIUGXRZ37U2EWL5A, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.243455536 (190h11m29.674535678s) [noise=true , address=79.127.201.215:4346 , piece_id=5HSE6YACVWJYCLNKJZDBP7FYX3BUZBK3PPVLH4B2SRQ7NDTI7BBQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.243478760 (190h11m29.674558881s) [noise=true , address=77.90.31.12:4605 , piece_id=T3V6PKE5OHGXYHQNUY7EOEEMAEE4VGJ6H2UUZ7Y5MSKHQ3CGGH7A, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.243773104 (190h11m29.674853235s) [noise=true , address=104.36.82.60:4361 , piece_id=MJOWHBEVPINP6TDPOTW5SOSZFFSIYOEJP24MMOC3A242TQAGTA4Q, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.243859078 (190h11m29.674939209s) [noise=true , address=104.36.82.34:4555 , piece_id=7VUV6GKFBXZXATSDHJXPDRIRGZCCEDVNVUTQHOOTG7M53ZQVOM5Q, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.244271377 (190h11m29.675351508s) [noise=true , address=104.36.82.23:4450 , piece_id=EY7DZQIHAKFTJFTQ2DTRUMJ6VEKQHINMJRP4ZDIJGUTTSJSAVLRQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.244780942 (190h11m29.675861073s) [noise=true , address=104.36.82.23:4446 , piece_id=XLQJN2HDJHWJCKZVCVYFZISVISBXQKMNMQR2DKZVDYOZ633UBBTA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.246339124 (190h11m29.677419255s) [noise=true , address=104.36.82.59:4336 , piece_id=KI4AGVFGFM2NA44727VSWGJ5EDVHZEPO53YA44PZO76GK7BU2OFA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.246968608 (190h11m29.678048739s) [noise=true , address=79.127.201.214:4320 , piece_id=DKDIIHZ246KRFSWW2HQBPUHQPMLBGFHM4CGILEKCHCBYW2WDGMGA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.248086416 (190h11m29.679166547s) [noise=true , address=45.91.116.10:4110 , piece_id=2HA3PQXJXSXTJ44OEBIXBBJAVLV7LTHNMKSLBXINDMQTNRPWOSNQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.249007910 (190h11m29.680088051s) [noise=true , address=45.91.116.12:4257 , piece_id=7TT3FBQWZV5BC4DTYBN5MSZTDN24IR4JQ7WNBPN5P3WK6ADLGKCA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.249398046 (190h11m29.680478187s) [noise=true , address=104.36.82.9:4035 , piece_id=YLTXFTAR4TUXAHNZQ4J33GP75AJRZ5SXCUGAZOUZR7HJ5GL47MIQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.249860161 (190h11m29.680940292s) [noise=true , address=77.90.31.11:4556 , piece_id=2KXEE6WQZCKZ2655POWT2GFUAN4XOMYHYWDHERSDP4LBHO5MIQTQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.250194720 (190h11m29.681274841s) [noise=true , address=104.36.82.44:4899 , piece_id=JX2EVIDRODAZP6UXTTKF3BALMPOOI76ZB45QFU5RBNWORJ4WJMZA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.250523189 (190h11m29.68160332s) [noise=true , address=104.36.82.5:4868 , piece_id=RTLPGUXXVRY4LQZUZRK3B6T3GCW74H3FGI74NMUVME6TUKQOYNVA, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.250919889 (190h11m29.68200001s) [noise=true , address=77.90.31.10:4545 , piece_id=JWHZHJEIJ2H6DAHRUWRJ3XHSFKDV7NLMBI5M2XXS54FYBCO6KI6A, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:35.251242596 (190h11m29.682322727s) [noise=true , address=77.90.31.19:4870 , piece_id=LTVYMCYJBCGLN5DN34HKFUNV2D2CDMVWPS4K2TVTDYIAWRLYIOOQ, seg_pos={0 0}, upload=197873]: Uploading piece... Aug 16 15:37:36.380933473 (190h11m30.812013604s) [upload=197873]: Next returned nil segment Aug 16 15:37:36.380950296 (190h11m30.812030417s) [upload=197873]: Waiting for error group managing segments... Aug 16 15:37:37.152013049 (190h11m31.58309318s) [noise=true , address=77.90.31.19:4870 , piece_id=LTVYMCYJBCGLN5DN34HKFUNV2D2CDMVWPS4K2TVTDYIAWRLYIOOQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:37.272443037 (190h11m31.703523178s) [noise=true , address=77.90.31.12:4605 , piece_id=T3V6PKE5OHGXYHQNUY7EOEEMAEE4VGJ6H2UUZ7Y5MSKHQ3CGGH7A, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:37.332155150 (190h11m31.763235281s) [noise=true , address=77.90.31.11:4556 , piece_id=2KXEE6WQZCKZ2655POWT2GFUAN4XOMYHYWDHERSDP4LBHO5MIQTQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:37.437287870 (190h11m31.868368001s) [noise=true , address=77.90.31.21:4882 , piece_id=TTG6ZYGQOJFR2P4K2NHOJ7DFDAZUFWCC7U64QIUGXRZ37U2EWL5A, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:38.060861079 (190h11m32.49194121s) [noise=true , address=45.91.116.10:4110 , piece_id=2HA3PQXJXSXTJ44OEBIXBBJAVLV7LTHNMKSLBXINDMQTNRPWOSNQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:39.696955884 (190h11m34.128036005s) [noise=true , address=77.90.31.10:4545 , piece_id=JWHZHJEIJ2H6DAHRUWRJ3XHSFKDV7NLMBI5M2XXS54FYBCO6KI6A, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:40.099331786 (190h11m34.530411917s) [noise=true , address=79.127.201.214:4320 , piece_id=DKDIIHZ246KRFSWW2HQBPUHQPMLBGFHM4CGILEKCHCBYW2WDGMGA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:41.144751060 (190h11m35.575831181s) [noise=true , address=79.127.201.215:4346 , piece_id=5HSE6YACVWJYCLNKJZDBP7FYX3BUZBK3PPVLH4B2SRQ7NDTI7BBQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:42.744852049 (190h11m37.17593218s) [noise=true , address=66.23.193.175:4555 , piece_id=IJBP3V6GT3WWW6GZIZTIV3E22LJXPA6NZIGF4EWSR4WAKV4ZW6IQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:37:45.224537976 (190h11m39.655618107s) [noise=true , address=45.91.116.12:4257 , piece_id=7TT3FBQWZV5BC4DTYBN5MSZTDN24IR4JQ7WNBPN5P3WK6ADLGKCA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: <nil> Aug 16 15:38:34.145217999 (190h12m28.57629812s) [noise=true , address=104.36.82.34:4555 , piece_id=7VUV6GKFBXZXATSDHJXPDRIRGZCCEDVNVUTQHOOTG7M53ZQVOM5Q, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:129JPGkrFLo95oqnbyr55FBPqPts6UPzX8VQpHf167o5FQV8WFD, address:104.36.82.34:4555): protocol: rpc: dial tcp 104.36.82.34:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145261692 (190h12m28.576341813s) [noise=true , address=104.36.82.23:4450 , piece_id=EY7DZQIHAKFTJFTQ2DTRUMJ6VEKQHINMJRP4ZDIJGUTTSJSAVLRQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms, address:104.36.82.23:4450): protocol: rpc: dial tcp 104.36.82.23:4450: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145298443 (190h12m28.576378574s) [noise=true , address=104.36.82.23:4446 , piece_id=XLQJN2HDJHWJCKZVCVYFZISVISBXQKMNMQR2DKZVDYOZ633UBBTA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 188.166.225.170:51686->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145330645 (190h12m28.576410776s) [noise=true , address=104.36.82.44:4899 , piece_id=JX2EVIDRODAZP6UXTTKF3BALMPOOI76ZB45QFU5RBNWORJ4WJMZA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: write tcp 188.166.225.170:39556->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145520909 (190h12m28.57660104s) [noise=true , address=104.36.82.59:4336 , piece_id=KI4AGVFGFM2NA44727VSWGJ5EDVHZEPO53YA44PZO76GK7BU2OFA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:12W6uY3xtjFp52F1Q3PwjX8Xb53NPcZxxf3KbxhRafAL6Prm4u5, address:104.36.82.59:4336): protocol: rpc: dial tcp 104.36.82.59:4336: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145553481 (190h12m28.576633612s) [noise=true , address=104.36.82.9:4035 , piece_id=YLTXFTAR4TUXAHNZQ4J33GP75AJRZ5SXCUGAZOUZR7HJ5GL47MIQ, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:1vucBTonz7hjJgo2Jy2rhpSryqK6ifrBtj34pAcqc8FA5nPyYp, address:104.36.82.9:4035): protocol: write tcp 188.166.225.170:52592->104.36.82.9:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145601132 (190h12m28.576681263s) [noise=true , address=104.36.82.5:4868 , piece_id=RTLPGUXXVRY4LQZUZRK3B6T3GCW74H3FGI74NMUVME6TUKQOYNVA, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145668912 (190h12m28.576749043s) [noise=true , address=104.36.82.60:4361 , piece_id=MJOWHBEVPINP6TDPOTW5SOSZFFSIYOEJP24MMOC3A242TQAGTA4Q, seg_pos={0 0}, upload=197873]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: rpc: dial tcp 104.36.82.60:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:38:34.145757832 (190h12m28.576837964s) [seg_pos={0 0}, upload=197873]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:38:34.373619870 (190h12m28.804700001s) [upload=197873]: Done uploading segments... Aug 16 15:38:34.373645268 (190h12m28.8047254s) [upload=197873]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230835629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.......X...X...X..................X.X..X.....X..............] duration: 6.801676865s FAIL (2025-08-16 15:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 6.801704547s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [........X...X...X..................X.X..X.....X.............] duration: 3.595373663s FAIL (2025-08-16 15:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 3.595391597s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................................X......X..XXXXXXXX] duration: 30.229408598s FAIL (2025-08-16 15:37:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:37:30.013843078 (190h11m24.444923209s) [upload=197870]: Starting upload Aug 16 15:37:30.013906890 (190h11m24.444987021s) [upload=197870]: Done starting upload Aug 16 15:37:30.014023373 (190h11m24.445103504s) [upload=197870]: Uploading segments... Aug 16 15:37:30.014130027 (190h11m24.445210158s) [upload=197870]: Got next segment. Inline: false Aug 16 15:37:30.014169352 (190h11m24.445249473s) [upload=197870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:37:30.014174031 (190h11m24.445254162s) [upload=197870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:37:30.258151716 (190h11m24.689231847s) [seg_pos={0 0}, upload=197870]: Begin upload segment... Aug 16 15:37:30.258187344 (190h11m24.689267475s) [seg_pos={0 0}, upload=197870]: Done begin upload segment. Aug 16 15:37:30.258205749 (190h11m24.68928588s) [noise=true , address=104.36.82.7:4910 , piece_id=F7YGFGA6GCYMC3THN2KXMQDYUPEYJZNBCGXY5BLDP5AEZ2FBKODQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.258268239 (190h11m24.68934838s) [noise=true , address=104.36.82.11:4114 , piece_id=CWD5EQ2ZISHO2JTTLIF4ZBMTEQM55U7OA4ICJR6QSMZDRDQEGSCQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.258553895 (190h11m24.689634016s) [noise=true , address=104.36.82.14:4223 , piece_id=2RA3GDSGFAIE27YJ77PQXP27V56UOXH7REOZ5IHMGGGCASH5M7AA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.258891672 (190h11m24.689971803s) [noise=true , address=104.36.82.15:4257 , piece_id=VK5MGWKANUTWR7RSN25FE5LSGFQ7QM5KHU6D5UGS7B3HVDJJGNQQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.259054994 (190h11m24.690135125s) [noise=true , address=104.36.82.12:4151 , piece_id=ZMPOJXLQV3XC3KQVCKAB7RZW5LA75GZHOXRQPXGT2KGMR2BPBSNA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.259337635 (190h11m24.690417766s) [noise=true , address=104.36.82.7:4909 , piece_id=N7N4RK2IY35XL3RR6GKMSWQHVTUNHNJUWYDA7S6J35GZFTD6JCXQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.259395496 (190h11m24.690475627s) [noise=true , address=104.36.82.4:4801 , piece_id=BAUTR2SCYSUS6FGMKYIYQPK2WJFDKAUGAJGK5GLLDUF2KI3VWEGA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.259668118 (190h11m24.690748249s) [noise=true , address=77.90.31.18:4806 , piece_id=3RKHB6PQD3735DVMET4ZODQ3PJMADL77FGJJLDXC7WTDTC3G5Z6A, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.259880354 (190h11m24.690960485s) [noise=true , address=77.90.31.11:4554 , piece_id=UPQMZPZPLUVYWUNPEO4TMGZSIWCRFNM7WKJFJUQMASCG4EE5JXZA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260124731 (190h11m24.691204852s) [noise=true , address=77.90.31.13:4628 , piece_id=2GDYGHJV3F6X4DEJ3EUJOQY6IQPDWSXCMB4FPSRLJDOZCQX6DKSA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260131224 (190h11m24.691211355s) [noise=true , address=104.36.82.6:4874 , piece_id=YFXKWFJJ6MLU3CNXI4GO37LWHUZOBQBZUJYKXUVISQRJVZNEBXKA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260387714 (190h11m24.691467845s) [noise=true , address=104.36.82.2:4731 , piece_id=6EYK4AKEUCYSGLN3KIRPUCZ7BSISU57MNG4ECFGSQR7PHXJUQH4Q, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260530948 (190h11m24.691611079s) [noise=true , address=104.36.82.16:4294 , piece_id=HPHPCZA624Y2K5OYUZGN7DGAEZEC3WQNMCUTLSQVMCNXQ2NINXHA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260594480 (190h11m24.691674611s) [noise=true , address=77.90.31.15:4697 , piece_id=Z566H5HN5CU657RBUUZHXTLVS4IOU5YOHGFVD77BGZEIUOFOZBPQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.260899113 (190h11m24.691979244s) [noise=true , address=104.36.82.10:4079 , piece_id=4GWJYEU4I2DJI5MBQEYTOXHRDMBC44UKXWW5K3KABWTZ4VSLVQGQ, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:30.261260484 (190h11m24.692340625s) [noise=true , address=45.91.116.5:4298 , piece_id=CMA7CADVMFZVVVLG4JZRUC55CVHTUOBSNOHZG443B5UAH4HMJIAA, seg_pos={0 0}, upload=197870]: Uploading piece... Aug 16 15:37:32.036565365 (190h11m26.467645496s) [upload=197870]: Next returned nil segment Aug 16 15:37:32.036582468 (190h11m26.467662599s) [upload=197870]: Waiting for error group managing segments... Aug 16 15:37:32.311462039 (190h11m26.74254217s) [noise=true , address=104.36.82.7:4910 , piece_id=F7YGFGA6GCYMC3THN2KXMQDYUPEYJZNBCGXY5BLDP5AEZ2FBKODQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.314829402 (190h11m26.745909533s) [noise=true , address=104.36.82.6:4874 , piece_id=YFXKWFJJ6MLU3CNXI4GO37LWHUZOBQBZUJYKXUVISQRJVZNEBXKA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.394740564 (190h11m26.825820695s) [noise=true , address=104.36.82.11:4114 , piece_id=CWD5EQ2ZISHO2JTTLIF4ZBMTEQM55U7OA4ICJR6QSMZDRDQEGSCQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.402048266 (190h11m26.833128397s) [noise=true , address=104.36.82.7:4909 , piece_id=N7N4RK2IY35XL3RR6GKMSWQHVTUNHNJUWYDA7S6J35GZFTD6JCXQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.456470536 (190h11m26.887550667s) [noise=true , address=77.90.31.13:4628 , piece_id=2GDYGHJV3F6X4DEJ3EUJOQY6IQPDWSXCMB4FPSRLJDOZCQX6DKSA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.458800794 (190h11m26.889880925s) [noise=true , address=104.36.82.12:4151 , piece_id=ZMPOJXLQV3XC3KQVCKAB7RZW5LA75GZHOXRQPXGT2KGMR2BPBSNA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.465120065 (190h11m26.896200196s) [noise=true , address=104.36.82.10:4079 , piece_id=4GWJYEU4I2DJI5MBQEYTOXHRDMBC44UKXWW5K3KABWTZ4VSLVQGQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.465547083 (190h11m26.896627214s) [noise=true , address=104.36.82.15:4257 , piece_id=VK5MGWKANUTWR7RSN25FE5LSGFQ7QM5KHU6D5UGS7B3HVDJJGNQQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.470754647 (190h11m26.901834778s) [noise=true , address=45.91.116.5:4298 , piece_id=CMA7CADVMFZVVVLG4JZRUC55CVHTUOBSNOHZG443B5UAH4HMJIAA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:32.551483864 (190h11m26.982563995s) [noise=true , address=77.90.31.11:4554 , piece_id=UPQMZPZPLUVYWUNPEO4TMGZSIWCRFNM7WKJFJUQMASCG4EE5JXZA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: <nil> Aug 16 15:37:59.364702438 (190h11m53.795782569s) [noise=true , address=77.90.31.15:4697 , piece_id=Z566H5HN5CU657RBUUZHXTLVS4IOU5YOHGFVD77BGZEIUOFOZBPQ, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:47392->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:59.364759928 (190h11m53.795840059s) [noise=true , address=104.36.82.14:4223 , piece_id=2RA3GDSGFAIE27YJ77PQXP27V56UOXH7REOZ5IHMGGGCASH5M7AA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:37:59.364797139 (190h11m53.79587727s) [noise=true , address=104.36.82.2:4731 , piece_id=6EYK4AKEUCYSGLN3KIRPUCZ7BSISU57MNG4ECFGSQR7PHXJUQH4Q, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:59.364819111 (190h11m53.795899242s) [noise=true , address=104.36.82.4:4801 , piece_id=BAUTR2SCYSUS6FGMKYIYQPK2WJFDKAUGAJGK5GLLDUF2KI3VWEGA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:59.364840773 (190h11m53.795920904s) [noise=true , address=104.36.82.16:4294 , piece_id=HPHPCZA624Y2K5OYUZGN7DGAEZEC3WQNMCUTLSQVMCNXQ2NINXHA, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:59.364956724 (190h11m53.796036856s) [noise=true , address=77.90.31.18:4806 , piece_id=3RKHB6PQD3735DVMET4ZODQ3PJMADL77FGJJLDXC7WTDTC3G5Z6A, seg_pos={0 0}, upload=197870]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53172->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:59.364981231 (190h11m53.796061352s) [seg_pos={0 0}, upload=197870]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:37:59.592494284 (190h11m54.023574455s) [upload=197870]: Done uploading segments... Aug 16 15:37:59.592526435 (190h11m54.023606586s) [upload=197870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.229425201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X...........X......X...X.X......X..X......X..X...XXXXXXXXX] duration: 1m0.645853526s FAIL (2025-08-16 15:37:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:36:40.947000115 (190h10m35.378080246s) [upload=197859]: Starting upload Aug 16 15:36:40.947044140 (190h10m35.378124261s) [upload=197859]: Done starting upload Aug 16 15:36:40.947140845 (190h10m35.378220976s) [upload=197859]: Uploading segments... Aug 16 15:36:40.947394971 (190h10m35.378475102s) [upload=197859]: Got next segment. Inline: false Aug 16 15:36:40.947440969 (190h10m35.37852113s) [upload=197859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:36:40.947445718 (190h10m35.378525849s) [upload=197859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:36:41.691421737 (190h10m36.122501858s) [seg_pos={0 0}, upload=197859]: Begin upload segment... Aug 16 15:36:41.691481522 (190h10m36.122561643s) [seg_pos={0 0}, upload=197859]: Done begin upload segment. Aug 16 15:36:41.691503293 (190h10m36.122583424s) [noise=true , address=66.23.193.175:4559 , piece_id=ZFAXVZUYYCDXBMUF74KV4FPKLCXIVET5MI4DM3W7B42YYGESD6AA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.691542349 (190h10m36.12262248s) [noise=true , address=77.90.31.18:4838 , piece_id=57F3TDBFYGBSGZEQRPAGYJ6PZF76OZBC5UKBHVJHOZ6WELCLUXKQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.691895214 (190h10m36.122975345s) [noise=true , address=104.36.82.3:4788 , piece_id=JOBE2RGH77FMWZTMWLC77ZUUUL3MDY5NP4P2M3NDL7GSGT7VYZWA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.692162024 (190h10m36.123242155s) [noise=true , address=77.90.31.13:4636 , piece_id=N4WE4GOVOTC3RGYLS6MPFVWMKZTBRWIJXHL6IHJKCGLHDWGFUVJA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.692463340 (190h10m36.123543471s) [noise=true , address=104.36.82.18:4389 , piece_id=IDR6UBZF5CEV3KFKKS26H4VVE3FZQQWP4FRZOILSXDJTA634XTIQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.692720192 (190h10m36.123800313s) [noise=true , address=104.36.82.59:4323 , piece_id=B4LZYBLALP7HBRTFMT5NPLA7SJ2W5AZNFBGUHHSGJHATJC66FSMQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.692958879 (190h10m36.12403901s) [noise=true , address=104.36.82.58:4297 , piece_id=5WSG5GVGIC7FZVHZPKUGKK4JJLAGOQSEKLHWUTJY4UYA3PCFYM3A, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.693160845 (190h10m36.124240976s) [noise=true , address=77.90.31.21:4888 , piece_id=AS6BGGKKRZA5CYNPIXTP7O2JG7ETQXOVTNJ67Z3JTS7SSZBXDDFQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.693372821 (190h10m36.124452952s) [noise=true , address=104.36.82.44:4897 , piece_id=VC5DHJGG6IC7ZPRXHANQAQBIWTPQGOTXECWRSRSHVUHD2GJW64AQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.693571341 (190h10m36.124651472s) [noise=true , address=104.36.82.35:4599 , piece_id=32W7BPE4O7JHOWP26XQRCEAHSFC7ONBX24I6SOWIRP4KFPKCX75A, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.694389897 (190h10m36.125470018s) [noise=true , address=104.36.82.23:4476 , piece_id=TNNDRQWDW2NJO64TEA4DNTMAQVH6G3RF5C5DC2BJXRIKUH73M7WQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.694789762 (190h10m36.125869893s) [noise=true , address=45.91.116.5:4328 , piece_id=GYKAI7LEYZTNYDV3JZWETCRCYW7VNIRZHOMX7RIMQHFMMCFNZTUQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.695069317 (190h10m36.126149448s) [noise=true , address=77.90.31.10:4526 , piece_id=GAYVWFV3O3XQCXXV7VVLXLKS6JRQ6LI5OSOVPJUJF2S52JLP7ELA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.695333963 (190h10m36.126414094s) [noise=true , address=104.36.82.60:4357 , piece_id=COYEZH7K5QV2GIZZBFDGRUDV634WM3EVMI7LS2UO2VC4KNXHK4NA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.695593260 (190h10m36.126673391s) [noise=true , address=79.127.201.215:4349 , piece_id=D4DB4LAB6RGM2S6YD2LVYATM4WFTQBWQWTH74URF27ODMXOS6IOQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.695857134 (190h10m36.126937255s) [noise=true , address=104.36.82.5:4855 , piece_id=YW3UZODT7BO2DPDCPD6LLNAAKLBIPDSELV6WJKMXUTCBSSWCXWTQ, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.696050364 (190h10m36.127130495s) [noise=true , address=79.127.220.101:4075 , piece_id=O3UGAOPDL3UCERGQ6LLTSOPEUB7OOOKKO4CA6Y4AD6LRATN5AS2Q, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:41.696257270 (190h10m36.127337401s) [noise=true , address=104.36.82.8:4975 , piece_id=5OIILZC7EIMILUZNPTGB752UTFUJ3HQVFP2AORM2XTGVXPQO7UEA, seg_pos={0 0}, upload=197859]: Uploading piece... Aug 16 15:36:42.969623349 (190h10m37.40070348s) [upload=197859]: Next returned nil segment Aug 16 15:36:42.969643177 (190h10m37.400723308s) [upload=197859]: Waiting for error group managing segments... Aug 16 15:36:44.449833855 (190h10m38.880913986s) [noise=true , address=77.90.31.10:4526 , piece_id=GAYVWFV3O3XQCXXV7VVLXLKS6JRQ6LI5OSOVPJUJF2S52JLP7ELA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:44.504482830 (190h10m38.935562951s) [noise=true , address=77.90.31.21:4888 , piece_id=AS6BGGKKRZA5CYNPIXTP7O2JG7ETQXOVTNJ67Z3JTS7SSZBXDDFQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:44.645404481 (190h10m39.076484612s) [noise=true , address=77.90.31.13:4636 , piece_id=N4WE4GOVOTC3RGYLS6MPFVWMKZTBRWIJXHL6IHJKCGLHDWGFUVJA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:44.665184718 (190h10m39.09626485s) [noise=true , address=45.91.116.5:4328 , piece_id=GYKAI7LEYZTNYDV3JZWETCRCYW7VNIRZHOMX7RIMQHFMMCFNZTUQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:45.110168134 (190h10m39.541248265s) [noise=true , address=79.127.220.101:4075 , piece_id=O3UGAOPDL3UCERGQ6LLTSOPEUB7OOOKKO4CA6Y4AD6LRATN5AS2Q, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:48.240917483 (190h10m42.671997604s) [noise=true , address=79.127.201.215:4349 , piece_id=D4DB4LAB6RGM2S6YD2LVYATM4WFTQBWQWTH74URF27ODMXOS6IOQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:36:50.949011541 (190h10m45.380091672s) [noise=true , address=66.23.193.175:4559 , piece_id=ZFAXVZUYYCDXBMUF74KV4FPKLCXIVET5MI4DM3W7B42YYGESD6AA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: <nil> Aug 16 15:37:40.045214988 (190h11m34.476295119s) [noise=true , address=104.36.82.59:4323 , piece_id=B4LZYBLALP7HBRTFMT5NPLA7SJ2W5AZNFBGUHHSGJHATJC66FSMQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:12HLr6Ha1ysWYbUJ8He7RofMFCz8qVuJMBFkfJgnTUVw1kmXckz, address:104.36.82.59:4323): protocol: rpc: dial tcp 104.36.82.59:4323: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045256969 (190h11m34.4763371s) [noise=true , address=104.36.82.44:4897 , piece_id=VC5DHJGG6IC7ZPRXHANQAQBIWTPQGOTXECWRSRSHVUHD2GJW64AQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045287507 (190h11m34.476367639s) [noise=true , address=104.36.82.23:4476 , piece_id=TNNDRQWDW2NJO64TEA4DNTMAQVH6G3RF5C5DC2BJXRIKUH73M7WQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1foWwyGzFenUicfY7cTU5dmdDJsvXGR3YXv1rf3QRXMUco59TV, address:104.36.82.23:4476): protocol: rpc: dial tcp 104.36.82.23:4476: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045315661 (190h11m34.476395792s) [noise=true , address=104.36.82.3:4788 , piece_id=JOBE2RGH77FMWZTMWLC77ZUUUL3MDY5NP4P2M3NDL7GSGT7VYZWA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: rpc: dial tcp 104.36.82.3:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045349145 (190h11m34.476429276s) [noise=true , address=104.36.82.35:4599 , piece_id=32W7BPE4O7JHOWP26XQRCEAHSFC7ONBX24I6SOWIRP4KFPKCX75A, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1nxKQjpzyhZsRnkffX1fwCXcTYiwHPwuNg4HWtpdzRnKQCtp6s, address:104.36.82.35:4599): protocol: rpc: dial tcp 104.36.82.35:4599: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045377229 (190h11m34.47645735s) [noise=true , address=104.36.82.60:4357 , piece_id=COYEZH7K5QV2GIZZBFDGRUDV634WM3EVMI7LS2UO2VC4KNXHK4NA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:17EmBobdmk16xm9girzx3asYwkhdCKKZCCPrBLwWTS4ujMdGmi, address:104.36.82.60:4357): protocol: rpc: dial tcp 104.36.82.60:4357: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045404030 (190h11m34.476484161s) [noise=true , address=104.36.82.8:4975 , piece_id=5OIILZC7EIMILUZNPTGB752UTFUJ3HQVFP2AORM2XTGVXPQO7UEA, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:104.36.82.8:4975): protocol: rpc: dial tcp 104.36.82.8:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045434729 (190h11m34.47651486s) [noise=true , address=104.36.82.18:4389 , piece_id=IDR6UBZF5CEV3KFKKS26H4VVE3FZQQWP4FRZOILSXDJTA634XTIQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1mMqCXPNzFx4cnu8Nb3gNKcJFJ4tGJjnkwY2EVPjUBnwNVcM9R, address:104.36.82.18:4389): protocol: rpc: dial tcp 104.36.82.18:4389: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045460959 (190h11m34.47654109s) [noise=true , address=104.36.82.58:4297 , piece_id=5WSG5GVGIC7FZVHZPKUGKK4JJLAGOQSEKLHWUTJY4UYA3PCFYM3A, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:12fVAGmLTkEZ2oauRnaMgyV9vvbRKZHj6eKW5vf2yZbPMDpRwfW, address:104.36.82.58:4297): protocol: rpc: dial tcp 104.36.82.58:4297: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045489353 (190h11m34.476569484s) [noise=true , address=104.36.82.5:4855 , piece_id=YW3UZODT7BO2DPDCPD6LLNAAKLBIPDSELV6WJKMXUTCBSSWCXWTQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:38756->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045518008 (190h11m34.476598139s) [noise=true , address=77.90.31.18:4838 , piece_id=57F3TDBFYGBSGZEQRPAGYJ6PZF76OZBC5UKBHVJHOZ6WELCLUXKQ, seg_pos={0 0}, upload=197859]: Done uploading piece. err: ecclient: upload failed (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, address:77.90.31.18:4838): protocol: read tcp 188.166.225.170:51588->77.90.31.18:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:40.045531064 (190h11m34.476611195s) [seg_pos={0 0}, upload=197859]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:37:40.689426737 (190h11m35.120506869s) [upload=197859]: Done uploading segments... Aug 16 15:37:40.689450513 (190h11m35.120530644s) [upload=197859]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.645863666s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.224122117s FAIL (2025-08-16 15:37:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== EBGZGA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.792392241 (190h10m29.223472372s) [noise=true , address=66.23.193.175:4556 , piece_id=VKBGZCJZZVRYJ75O6H6O3P4HBOAM5NXNP76T6GEJGLJXJPUKOOQA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.792529073 (190h10m29.223609204s) [noise=true , address=77.90.31.10:4538 , piece_id=FKPCYP5TPRUK34U35A6KCNBNWP4CIYLGEYEYFSDIYPJKLVXG4MCA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.792751368 (190h10m29.223831499s) [noise=true , address=79.127.220.101:4061 , piece_id=S35IG6LENW7J2PZXKEQCCOBCHENDO6PKBVIS2TMWLRVKQCIEDXPA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.792827543 (190h10m29.223907674s) [noise=true , address=104.36.82.36:4624 , piece_id=NERHI27HUBKKTKC6ABCVGP3LGCDLSEFVH3M6RBR4AXXNJZ6LA45Q, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.793105987 (190h10m29.224186118s) [noise=true , address=45.91.116.8:4069 , piece_id=GWZEIHRD7FKRN2N66D575I23V3YHU7TOD64XI4VQPJQXBDTCJO7A, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.793209013 (190h10m29.224289134s) [noise=true , address=77.90.31.12:4619 , piece_id=N66KNANKBS6OGBENITSFRDHSSQHQOKI6MVL3MPMMSC2RQEGM7NWQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.793415579 (190h10m29.2244957s) [noise=true , address=104.36.82.44:4878 , piece_id=EILLTB2I3E726PAMLT7ENLMIVCI23XDB6YOQPU4B4HD2523Q6VLA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.793792250 (190h10m29.224872381s) [noise=true , address=156.146.43.225:4457 , piece_id=PLPN5P5DT6KUOESTCI5KHSNQ3X5YCR7CWDMU4QADBON52WAFWYKQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.794197995 (190h10m29.225278126s) [noise=true , address=104.36.82.60:4340 , piece_id=U6UVCNN5J72AVZZPMLJRPBFKRSSEDJIHJ6VLXPIYUW7OD6AS3FOA, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.794534530 (190h10m29.225614661s) [noise=true , address=79.127.201.215:4354 , piece_id=DK7RPPBELMMGM56SYCAJAZES6SEFLVUXN3YYQNNL4BJGA2544SZQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.794750674 (190h10m29.225830795s) [noise=true , address=77.90.31.13:4635 , piece_id=PWWP64X7UED7X22ZCTSAQJDMYR2WZ7X6GXZQIFIFC3E6YT23MROQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.794946528 (190h10m29.226026659s) [noise=true , address=77.90.31.18:4827 , piece_id=M3BB23CILVACNFAIY4OB762RFWLEQKSU3ZJE4FJ6UD3DZS6RWVGQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.795146190 (190h10m29.226226321s) [noise=true , address=104.36.82.4:4808 , piece_id=4GXTKVIZGTG3EX4UZRV6QZTROT2ZC27CJ6TSXJ5P4DENWP5HF5VQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.795381661 (190h10m29.226461792s) [noise=true , address=104.36.82.13:4195 , piece_id=BZSWAD3UDPN5KTHKKLGALBGMXL7CBWIMYZQZSL7OXIV2SKR5HUYQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.795596922 (190h10m29.226677063s) [noise=true , address=79.127.201.214:4320 , piece_id=ROOW6ERETCK7X3AL634XV3LT4MMPTWPCUM3C44TZ45QOI6GFFFEQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:34.795788048 (190h10m29.226868169s) [noise=true , address=45.91.116.8:4058 , piece_id=3MOSZKQOLY7KNLAVNJ747U7EQQ2WCOIGLROB23NA53GQVMIXWWYQ, seg_pos={0 0}, upload=197856]: Uploading piece... Aug 16 15:36:35.977893218 (190h10m30.408973349s) [upload=197856]: Next returned nil segment Aug 16 15:36:35.977911582 (190h10m30.408991713s) [upload=197856]: Waiting for error group managing segments... Aug 16 15:36:36.844025401 (190h10m31.275105522s) [noise=true , address=45.91.116.8:4069 , piece_id=GWZEIHRD7FKRN2N66D575I23V3YHU7TOD64XI4VQPJQXBDTCJO7A, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:36.875532307 (190h10m31.306612438s) [noise=true , address=77.90.31.10:4538 , piece_id=FKPCYP5TPRUK34U35A6KCNBNWP4CIYLGEYEYFSDIYPJKLVXG4MCA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:36.928990024 (190h10m31.360070155s) [noise=true , address=77.90.31.13:4635 , piece_id=PWWP64X7UED7X22ZCTSAQJDMYR2WZ7X6GXZQIFIFC3E6YT23MROQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:37.015267378 (190h10m31.446347509s) [noise=true , address=156.146.43.225:4457 , piece_id=PLPN5P5DT6KUOESTCI5KHSNQ3X5YCR7CWDMU4QADBON52WAFWYKQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:37.036176845 (190h10m31.467256976s) [noise=true , address=77.90.31.12:4619 , piece_id=N66KNANKBS6OGBENITSFRDHSSQHQOKI6MVL3MPMMSC2RQEGM7NWQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:37.767829165 (190h10m32.198909286s) [noise=true , address=45.91.116.8:4058 , piece_id=3MOSZKQOLY7KNLAVNJ747U7EQQ2WCOIGLROB23NA53GQVMIXWWYQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:37.964008777 (190h10m32.395088898s) [noise=true , address=79.127.201.214:4320 , piece_id=ROOW6ERETCK7X3AL634XV3LT4MMPTWPCUM3C44TZ45QOI6GFFFEQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:39.799863806 (190h10m34.230943937s) [noise=true , address=79.127.220.101:4061 , piece_id=S35IG6LENW7J2PZXKEQCCOBCHENDO6PKBVIS2TMWLRVKQCIEDXPA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:42.640011537 (190h10m37.071091668s) [noise=true , address=79.127.201.215:4354 , piece_id=DK7RPPBELMMGM56SYCAJAZES6SEFLVUXN3YYQNNL4BJGA2544SZQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:36:43.343920301 (190h10m37.775000432s) [noise=true , address=66.23.193.175:4556 , piece_id=VKBGZCJZZVRYJ75O6H6O3P4HBOAM5NXNP76T6GEJGLJXJPUKOOQA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: <nil> Aug 16 15:37:33.700011215 (190h11m28.131091346s) [noise=true , address=104.36.82.44:4878 , piece_id=EILLTB2I3E726PAMLT7ENLMIVCI23XDB6YOQPU4B4HD2523Q6VLA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1wX9xRmyv3KWVyJhFmgBt9uYqDU6U768dqHagKZeKBSEnFup5y, address:104.36.82.44:4878): protocol: write tcp 188.166.225.170:58802->104.36.82.44:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700057343 (190h11m28.131137474s) [noise=true , address=104.36.82.23:4462 , piece_id=WIDQZF25E6RWH3E5I4LBU3INZXEFNSATH2X7AV2EBDVMSBCMSLBA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700117588 (190h11m28.131197719s) [noise=true , address=104.36.82.60:4340 , piece_id=U6UVCNN5J72AVZZPMLJRPBFKRSSEDJIHJ6VLXPIYUW7OD6AS3FOA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: write tcp 188.166.225.170:58868->104.36.82.60:4340: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700173235 (190h11m28.131253366s) [noise=true , address=104.36.82.36:4624 , piece_id=NERHI27HUBKKTKC6ABCVGP3LGCDLSEFVH3M6RBR4AXXNJZ6LA45Q, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: write tcp 188.166.225.170:39690->104.36.82.36:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700207321 (190h11m28.131287452s) [noise=true , address=77.90.31.18:4827 , piece_id=M3BB23CILVACNFAIY4OB762RFWLEQKSU3ZJE4FJ6UD3DZS6RWVGQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, address:77.90.31.18:4827): protocol: write tcp 188.166.225.170:55312->77.90.31.18:4827: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700242117 (190h11m28.131322248s) [noise=true , address=104.36.82.4:4808 , piece_id=4GXTKVIZGTG3EX4UZRV6QZTROT2ZC27CJ6TSXJ5P4DENWP5HF5VQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1fbQCaGY9ZssBzyNusnhyY994e8ytvW4sQDQRt7q2tdGrg6ps3, address:104.36.82.4:4808): protocol: write tcp 188.166.225.170:36700->104.36.82.4:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700263849 (190h11m28.13134398s) [noise=true , address=104.36.82.39:4748 , piece_id=POKEZHHJTKPKDRLLZWSPW7WKKKSPOGIAUST4BFSGVNOZTTEBGZGA, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:1aZ4HiQxxV3qYJLYViMi7me9mRnPPBgPihW8fwzeMTKXnwbUb9, address:104.36.82.39:4748): protocol: write tcp 188.166.225.170:46422->104.36.82.39:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700275120 (190h11m28.131355241s) [noise=true , address=104.36.82.13:4195 , piece_id=BZSWAD3UDPN5KTHKKLGALBGMXL7CBWIMYZQZSL7OXIV2SKR5HUYQ, seg_pos={0 0}, upload=197856]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:104.36.82.13:4195): protocol: write tcp 188.166.225.170:35566->104.36.82.13:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:37:33.700346848 (190h11m28.131426969s) [seg_pos={0 0}, upload=197856]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:37:33.922513087 (190h11m28.353593218s) [upload=197856]: Done uploading segments... Aug 16 15:37:33.922537695 (190h11m28.353617826s) [upload=197856]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.224135193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X...X..................X.X..X.....X............] duration: 1.793658329s FAIL (2025-08-16 15:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 1.793690699s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................X......X..XXXXXXX] duration: 30.236199178s FAIL (2025-08-16 15:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:36:30.028484723 (190h10m24.459564854s) [upload=197855]: Starting upload Aug 16 15:36:30.028541592 (190h10m24.459621723s) [upload=197855]: Done starting upload Aug 16 15:36:30.028640621 (190h10m24.459720752s) [upload=197855]: Uploading segments... Aug 16 15:36:30.028762063 (190h10m24.459842184s) [upload=197855]: Got next segment. Inline: false Aug 16 15:36:30.028794485 (190h10m24.459874616s) [upload=197855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:36:30.028798503 (190h10m24.459878634s) [upload=197855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:36:30.343565651 (190h10m24.774645772s) [seg_pos={0 0}, upload=197855]: Begin upload segment... Aug 16 15:36:30.343605737 (190h10m24.774685858s) [seg_pos={0 0}, upload=197855]: Done begin upload segment. Aug 16 15:36:30.343625385 (190h10m24.774705516s) [noise=true , address=104.36.82.13:4186 , piece_id=RP3NWK6435NXP2R3IJTK6N35WW3TE2USE2UM26P3SAJAAMKCR3KQ, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.343689689 (190h10m24.77476982s) [noise=true , address=45.91.116.8:4045 , piece_id=63UYPPF3RMC6HNJGLSC46ZOR33HMUCLM3EIZSVQNNPLWLSX5NR4Q, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344078553 (190h10m24.775158674s) [noise=true , address=104.36.82.11:4115 , piece_id=AFC4LTHYXEIVAYDDMNNVXHRFBBKGYQFPYNSKLB6YYUPGREJQR5TA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344233118 (190h10m24.775313249s) [noise=true , address=104.36.82.14:4223 , piece_id=ME77IIQH6XAZCNSPX2WI7WAKPWNYUL4NRSVTYQMEQVKCLS5FEVSQ, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344363748 (190h10m24.775443879s) [noise=true , address=45.91.116.9:4263 , piece_id=QIQJ6TKSXILAA62V6CLQRJM3U2UF6ZL5ZKJ7JEMOROLCWVPCAN3A, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344560504 (190h10m24.775640625s) [noise=true , address=104.36.82.14:4222 , piece_id=G2QFLMFUWUU7PSGY4IXOBHYDPXQHYWNHJOF4R7ZKKJX5MT4BL5AA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344616702 (190h10m24.775696833s) [noise=true , address=104.36.82.6:4873 , piece_id=OFSNNE6KLRLMPUVRQKJVANG7XHCRD65MNQTXYESFWN3UOQXPDA5A, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344818759 (190h10m24.77589889s) [noise=true , address=104.36.82.10:4077 , piece_id=KA7ZDEW2YS6VT7OR7OCZWVXQQJEXNRX37YOVWB7EIJNCJHPGYFYA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.344865908 (190h10m24.77594604s) [noise=true , address=45.91.116.12:4227 , piece_id=GPUNE2X5EFWT76ZPE2Q633XMQRIOJQNJM6GAD2R5SD2CMSAGFYCA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345093585 (190h10m24.776173716s) [noise=true , address=77.90.31.15:4699 , piece_id=B6ZKGDG2JVWZPEJAPNDNDBE6E22W24A6HRZOJ3OVTND5HC3NLO4Q, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345125957 (190h10m24.776206088s) [noise=true , address=104.36.82.16:4295 , piece_id=275QJZ5LYUVLJ5IEHWPZZR63RWF26D3UK3CAP3LVBINRLVXVHEOA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345336309 (190h10m24.77641644s) [noise=true , address=77.90.31.16:4733 , piece_id=HXQOGLLT3SDPQJBREVXLJLMTRBC263XLPQIFTO727BBP5YGQMZQA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345355115 (190h10m24.776435246s) [noise=true , address=104.36.82.6:4874 , piece_id=QG7ORTZPTIF2SZM74VBCITF35KAIY65665G2XNDSKMRJZJK4QWTQ, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345695256 (190h10m24.776775407s) [noise=true , address=104.36.82.15:4258 , piece_id=CTJMVLJ7NLBKDYLQ2PRX4W4PXLBNJ6BWZMJNFD4AT4YTY5GW3C2A, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.345952237 (190h10m24.777032368s) [noise=true , address=45.91.116.8:4047 , piece_id=XDIGHR2SD27V22VC3YTM7RX5JCI5MKG7PE64M6R3GVWSESVC3HRQ, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:30.346201114 (190h10m24.777281245s) [noise=true , address=77.90.31.13:4626 , piece_id=LFKP4M7PZT7H7I3KJZXBPRQ7SRBWCHUH6YHJACI7TQZSJQ5RM5OA, seg_pos={0 0}, upload=197855]: Uploading piece... Aug 16 15:36:32.083679910 (190h10m26.514760041s) [upload=197855]: Next returned nil segment Aug 16 15:36:32.083700929 (190h10m26.51478106s) [upload=197855]: Waiting for error group managing segments... Aug 16 15:36:32.371179894 (190h10m26.802260025s) [noise=true , address=45.91.116.8:4045 , piece_id=63UYPPF3RMC6HNJGLSC46ZOR33HMUCLM3EIZSVQNNPLWLSX5NR4Q, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.413887572 (190h10m26.844967703s) [noise=true , address=45.91.116.9:4263 , piece_id=QIQJ6TKSXILAA62V6CLQRJM3U2UF6ZL5ZKJ7JEMOROLCWVPCAN3A, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.421865877 (190h10m26.852946008s) [noise=true , address=77.90.31.15:4699 , piece_id=B6ZKGDG2JVWZPEJAPNDNDBE6E22W24A6HRZOJ3OVTND5HC3NLO4Q, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.439743695 (190h10m26.870823826s) [noise=true , address=104.36.82.11:4115 , piece_id=AFC4LTHYXEIVAYDDMNNVXHRFBBKGYQFPYNSKLB6YYUPGREJQR5TA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.450175394 (190h10m26.881255525s) [noise=true , address=104.36.82.6:4874 , piece_id=QG7ORTZPTIF2SZM74VBCITF35KAIY65665G2XNDSKMRJZJK4QWTQ, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.451812978 (190h10m26.882893099s) [noise=true , address=104.36.82.15:4258 , piece_id=CTJMVLJ7NLBKDYLQ2PRX4W4PXLBNJ6BWZMJNFD4AT4YTY5GW3C2A, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.455502056 (190h10m26.886582187s) [noise=true , address=104.36.82.6:4873 , piece_id=OFSNNE6KLRLMPUVRQKJVANG7XHCRD65MNQTXYESFWN3UOQXPDA5A, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.572970802 (190h10m27.004050933s) [noise=true , address=104.36.82.10:4077 , piece_id=KA7ZDEW2YS6VT7OR7OCZWVXQQJEXNRX37YOVWB7EIJNCJHPGYFYA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.618277292 (190h10m27.049357423s) [noise=true , address=45.91.116.8:4047 , piece_id=XDIGHR2SD27V22VC3YTM7RX5JCI5MKG7PE64M6R3GVWSESVC3HRQ, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.674111736 (190h10m27.105191867s) [noise=true , address=77.90.31.13:4626 , piece_id=LFKP4M7PZT7H7I3KJZXBPRQ7SRBWCHUH6YHJACI7TQZSJQ5RM5OA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:32.738627752 (190h10m27.169707893s) [noise=true , address=45.91.116.12:4227 , piece_id=GPUNE2X5EFWT76ZPE2Q633XMQRIOJQNJM6GAD2R5SD2CMSAGFYCA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: <nil> Aug 16 15:36:59.384642893 (190h10m53.815723024s) [noise=true , address=104.36.82.14:4222 , piece_id=G2QFLMFUWUU7PSGY4IXOBHYDPXQHYWNHJOF4R7ZKKJX5MT4BL5AA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op Aug 16 15:36:59.384693570 (190h10m53.815773701s) [noise=true , address=104.36.82.14:4223 , piece_id=ME77IIQH6XAZCNSPX2WI7WAKPWNYUL4NRSVTYQMEQVKCLS5FEVSQ, seg_pos={0 0}, upload=197855]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:59.384731332 (190h10m53.815811463s) [noise=true , address=104.36.82.16:4295 , piece_id=275QJZ5LYUVLJ5IEHWPZZR63RWF26D3UK3CAP3LVBINRLVXVHEOA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:35870->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:59.384758383 (190h10m53.815838514s) [noise=true , address=104.36.82.13:4186 , piece_id=RP3NWK6435NXP2R3IJTK6N35WW3TE2USE2UM26P3SAJAAMKCR3KQ, seg_pos={0 0}, upload=197855]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:59.384777220 (190h10m53.815857341s) [noise=true , address=77.90.31.16:4733 , piece_id=HXQOGLLT3SDPQJBREVXLJLMTRBC263XLPQIFTO727BBP5YGQMZQA, seg_pos={0 0}, upload=197855]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:59.384801256 (190h10m53.815881377s) [seg_pos={0 0}, upload=197855]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:36:59.618810030 (190h10m54.049890161s) [upload=197855]: Done uploading segments... Aug 16 15:36:59.618840879 (190h10m54.04992101s) [upload=197855]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.236219286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X...X..................X.X..X.....X...........] duration: 2.021682855s FAIL (2025-08-16 15:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 2.021709885s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X...........X......X...X.X......X..X......X..X...XXXXXXXX] duration: 1m1.586919573s FAIL (2025-08-16 15:36:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:35:38.897380138 (190h9m33.328460269s) [upload=197839]: Starting upload Aug 16 15:35:38.897423340 (190h9m33.328503461s) [upload=197839]: Done starting upload Aug 16 15:35:38.897554871 (190h9m33.328635003s) [upload=197839]: Uploading segments... Aug 16 15:35:38.897667517 (190h9m33.328747638s) [upload=197839]: Got next segment. Inline: false Aug 16 15:35:38.897699930 (190h9m33.328780061s) [upload=197839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:35:38.897703927 (190h9m33.328784048s) [upload=197839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:35:40.002829252 (190h9m34.433909383s) [seg_pos={0 0}, upload=197839]: Begin upload segment... Aug 16 15:35:40.002900770 (190h9m34.433980921s) [seg_pos={0 0}, upload=197839]: Done begin upload segment. Aug 16 15:35:40.002936237 (190h9m34.434016368s) [noise=true , address=77.90.31.10:4541 , piece_id=SEP2WLNBTASNQCL4FTQVXHW5BMNQXJQP3YPHOY24X7EJCRLF3GLA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.002965192 (190h9m34.434045344s) [noise=true , address=45.91.116.6:4985 , piece_id=WGZV4BWDGEO42KVPU3YY7OAPUAR6JXGULVEVWUSU3F5IN6VBMURQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.003380507 (190h9m34.434460628s) [noise=true , address=104.36.82.36:4642 , piece_id=CISOHSMDP5M77BADWO6NEF2UCLJU7DAT4T3U6V77SFXQE2IRCWCQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.003581070 (190h9m34.434661202s) [noise=true , address=104.36.82.59:4311 , piece_id=ROH7L6QPADC5EA4XWQH5EK22VNMSGWBSHLEV7MZFWIQWBCX5VHKA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.004059506 (190h9m34.435139638s) [noise=true , address=104.36.82.5:4841 , piece_id=C6UBSCMQSIPL7HH3PF7GTPKFY7MRK7RCOVKVBWXAXGWC75YD5CMQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.004402673 (190h9m34.435482814s) [noise=true , address=77.90.31.14:4667 , piece_id=WDHU2I6VNLUBQBPWVMZDLA3Q2ASTXELTYVZ5MPDHAF2OWD4M7M6A, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.004881089 (190h9m34.43596122s) [noise=true , address=66.23.193.175:4556 , piece_id=MPHV2Y3OJ7YZIFIY4QABBVUIUPWBXL6U44PFIBU5BKLJKY2IMEVQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.005163158 (190h9m34.436243289s) [noise=true , address=104.36.82.51:4068 , piece_id=KLZFHHHB657G6KSG3ERWNNNEBX74VYZGHG7BY24SAPN5URKYFIXQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.005400563 (190h9m34.436480694s) [noise=true , address=45.91.116.5:4307 , piece_id=XO2QEI6PVQ5WGKOGEJMI5YGPEXCHJF3X4XD2G4IKIZSJZP3SVNOQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.005647625 (190h9m34.436727756s) [noise=true , address=104.36.82.39:4738 , piece_id=ZQUFQ6WHLH4MS62MWP7UWYQZGRVK2GDPWV6PFWBXT32Y6SYTNDMA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.005896812 (190h9m34.436976943s) [noise=true , address=104.36.82.23:4477 , piece_id=YF2GOXCYO6BHHNC62OPHD63TSCFKOCELIKRDVLMX56UANHKKRZ2Q, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.006107926 (190h9m34.437188047s) [noise=true , address=79.127.205.235:4689 , piece_id=C5FRBZAMUSDMPWZE7YLG67ZZFCKVCMB47FC5VQXWLSEFMVMWHGYQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.006311806 (190h9m34.437391937s) [noise=true , address=45.91.116.13:4178 , piece_id=BWTN3XFO26VI2J6WXD7F5RQRU3E6PXFMX2HSSWQV2JOECQD6K47Q, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.006512149 (190h9m34.43759227s) [noise=true , address=104.36.82.17:4360 , piece_id=LUGWUPUBGA53FRKQYAE77AMVJ5O5KPE3QOCGAK73SDN3IO46RUBA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.006706461 (190h9m34.437786592s) [noise=true , address=104.36.82.34:4559 , piece_id=CNUYX7DXE5XMXQIPNT3DKMQU2VQBNJVFWNKGSNXX4ZSUJ7EVO74A, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.006898318 (190h9m34.437978449s) [noise=true , address=104.36.82.44:4873 , piece_id=D2GXNYFCT62UZQROARKAXZITTXKG7QRNX4N7KB5HFKLUAQQ4WHEQ, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.007102719 (190h9m34.43818285s) [noise=true , address=77.90.31.11:4563 , piece_id=JHITIRAJZZ2UKPPI6YTGN4HTYADKGJMBK3EOJI6OFB4LAD6HZ7HA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.007287714 (190h9m34.438367845s) [noise=true , address=79.127.201.215:4348 , piece_id=PKXXCSUNSM4HRDGQTTXIF275MADSUHJKF2EXSK2M5FDQC5KGTYWA, seg_pos={0 0}, upload=197839]: Uploading piece... Aug 16 15:35:40.769541385 (190h9m35.200621516s) [noise=true , address=79.127.205.235:4689 , piece_id=C5FRBZAMUSDMPWZE7YLG67ZZFCKVCMB47FC5VQXWLSEFMVMWHGYQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12TFdxckSXAWn5yxsjR8XD97Rj6bmbteeEjrQXx7ko16AXnXE91, address:79.127.205.235:4689): protocol: storage node overloaded, request limit: 2000 Aug 16 15:35:41.170612637 (190h9m35.601692768s) [upload=197839]: Next returned nil segment Aug 16 15:35:41.170633337 (190h9m35.601713468s) [upload=197839]: Waiting for error group managing segments... Aug 16 15:35:43.271495474 (190h9m37.702575605s) [noise=true , address=104.36.82.5:4841 , piece_id=C6UBSCMQSIPL7HH3PF7GTPKFY7MRK7RCOVKVBWXAXGWC75YD5CMQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: write tcp 188.166.225.170:43482->104.36.82.5:4841: write: connection refused; conn closed: op Aug 16 15:35:43.702187503 (190h9m38.133267634s) [noise=true , address=45.91.116.6:4985 , piece_id=WGZV4BWDGEO42KVPU3YY7OAPUAR6JXGULVEVWUSU3F5IN6VBMURQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: storage node overloaded, request limit: 2000 Aug 16 15:35:48.721090450 (190h9m43.152170581s) [noise=true , address=77.90.31.14:4667 , piece_id=WDHU2I6VNLUBQBPWVMZDLA3Q2ASTXELTYVZ5MPDHAF2OWD4M7M6A, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:35:51.155586443 (190h9m45.586666574s) [noise=true , address=66.23.193.175:4556 , piece_id=MPHV2Y3OJ7YZIFIY4QABBVUIUPWBXL6U44PFIBU5BKLJKY2IMEVQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:35:51.189270133 (190h9m45.620350264s) [noise=true , address=79.127.201.215:4348 , piece_id=PKXXCSUNSM4HRDGQTTXIF275MADSUHJKF2EXSK2M5FDQC5KGTYWA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:35:52.146132541 (190h9m46.577212672s) [noise=true , address=45.91.116.13:4178 , piece_id=BWTN3XFO26VI2J6WXD7F5RQRU3E6PXFMX2HSSWQV2JOECQD6K47Q, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:35:55.053305879 (190h9m49.48438604s) [noise=true , address=45.91.116.5:4307 , piece_id=XO2QEI6PVQ5WGKOGEJMI5YGPEXCHJF3X4XD2G4IKIZSJZP3SVNOQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:35:57.743389686 (190h9m52.174469817s) [noise=true , address=77.90.31.10:4541 , piece_id=SEP2WLNBTASNQCL4FTQVXHW5BMNQXJQP3YPHOY24X7EJCRLF3GLA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:36:00.460470519 (190h9m54.89155065s) [noise=true , address=77.90.31.11:4563 , piece_id=JHITIRAJZZ2UKPPI6YTGN4HTYADKGJMBK3EOJI6OFB4LAD6HZ7HA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: <nil> Aug 16 15:36:38.230783435 (190h10m32.661863556s) [noise=true , address=104.36.82.17:4360 , piece_id=LUGWUPUBGA53FRKQYAE77AMVJ5O5KPE3QOCGAK73SDN3IO46RUBA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12wdxbyP46e11Z1Bp9poBN4WGLPwAD2U2Kydv8c5ozL63DPvxvc, address:104.36.82.17:4360): protocol: write tcp 188.166.225.170:42554->104.36.82.17:4360: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.230831578 (190h10m32.661911699s) [noise=true , address=104.36.82.51:4068 , piece_id=KLZFHHHB657G6KSG3ERWNNNEBX74VYZGHG7BY24SAPN5URKYFIXQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: write tcp 188.166.225.170:59122->104.36.82.51:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.230857747 (190h10m32.661937878s) [noise=true , address=104.36.82.44:4873 , piece_id=D2GXNYFCT62UZQROARKAXZITTXKG7QRNX4N7KB5HFKLUAQQ4WHEQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: write tcp 188.166.225.170:36278->104.36.82.44:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.230964722 (190h10m32.662044853s) [noise=true , address=104.36.82.39:4738 , piece_id=ZQUFQ6WHLH4MS62MWP7UWYQZGRVK2GDPWV6PFWBXT32Y6SYTNDMA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: rpc: dial tcp 104.36.82.39:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.231020368 (190h10m32.662100499s) [noise=true , address=104.36.82.59:4311 , piece_id=ROH7L6QPADC5EA4XWQH5EK22VNMSGWBSHLEV7MZFWIQWBCX5VHKA, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12bowY9sNioETZodrJQZF4P9evfk5fv82gT8nTEq3kvjMYi4buX, address:104.36.82.59:4311): protocol: write tcp 188.166.225.170:40298->104.36.82.59:4311: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.231026240 (190h10m32.662106371s) [noise=true , address=104.36.82.23:4477 , piece_id=YF2GOXCYO6BHHNC62OPHD63TSCFKOCELIKRDVLMX56UANHKKRZ2Q, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf, address:104.36.82.23:4477): protocol: write tcp 188.166.225.170:38282->104.36.82.23:4477: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.231050737 (190h10m32.662130868s) [noise=true , address=104.36.82.36:4642 , piece_id=CISOHSMDP5M77BADWO6NEF2UCLJU7DAT4T3U6V77SFXQE2IRCWCQ, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L, address:104.36.82.36:4642): protocol: write tcp 188.166.225.170:34530->104.36.82.36:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.231057239 (190h10m32.66213736s) [noise=true , address=104.36.82.34:4559 , piece_id=CNUYX7DXE5XMXQIPNT3DKMQU2VQBNJVFWNKGSNXX4ZSUJ7EVO74A, seg_pos={0 0}, upload=197839]: Done uploading piece. err: ecclient: upload failed (node:1XXGQfMeXYCUuSh5TUe8pTwwd19wn68r9buS11N8aJdx6vZysf, address:104.36.82.34:4559): protocol: write tcp 188.166.225.170:39192->104.36.82.34:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:38.231075193 (190h10m32.662155324s) [seg_pos={0 0}, upload=197839]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:36:39.816082959 (190h10m34.247163091s) [upload=197839]: Done uploading segments... Aug 16 15:36:39.816109751 (190h10m34.247189882s) [upload=197839]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m1.586931165s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.232985875s FAIL (2025-08-16 15:36:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== NENKZNE43U6Q6F4KIP5C3XPWLX7CLFW2G7BON2GQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.347624390 (190h9m28.778704511s) [noise=true , address=104.36.82.35:4585 , piece_id=5K2GZ7HQHPFOEEEXNGSDNGFW77IH3FKSJLMQLWIGOEVDAUOTKBBQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.347865160 (190h9m28.778945292s) [noise=true , address=79.127.220.101:4061 , piece_id=AG5IA6YX742WBIJWBAAZVKGZOXLFP5IFM33NY7S6TQLSPR6JMIQA, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.348092536 (190h9m28.779172667s) [noise=true , address=104.36.82.9:4025 , piece_id=YQFBM7BQH3EB7PPBTZRCPMWCRZEQUG5N2IX6WYI6URFJSYOG5U4A, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.348363515 (190h9m28.779443646s) [noise=true , address=104.36.82.39:4759 , piece_id=GTAZTKTF4W2ILKHDXLPYNYFNRDZZIRIXDDSXTJWYEEPDNAWO764A, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.348552747 (190h9m28.779632878s) [noise=true , address=104.36.82.5:4848 , piece_id=SUWUAU66F47ONBE5EZGCYHCK4KH2ZATW25DO2GD2YPJ2SYNBPHYA, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.348748722 (190h9m28.779828853s) [noise=true , address=104.36.82.58:4290 , piece_id=MXU4EOS5XQFFHBQMVCUYTVPH4P4Z24G62A26AZ5SSKNFG4NHZYWQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.348946219 (190h9m28.78002635s) [noise=true , address=104.36.82.18:4383 , piece_id=MKDDFLJ6QLR7HXL2QMZ3IFRQPG7J6OWGTC3KLF63IC7IFRJ3OZXQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.349130011 (190h9m28.780210132s) [noise=true , address=104.36.82.4:4816 , piece_id=VSCXL6AXY5AXOVDMZNFGSIRZ72CR2MINZLJMAXNXZWSA45UXURIQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.349312720 (190h9m28.780392851s) [noise=true , address=77.90.31.13:4632 , piece_id=ZN2MD6ROTXBYLIF7JGURINOFO35WWEWHDG36ONNOKEQJEQPLMRTA, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.349493547 (190h9m28.780573678s) [noise=true , address=156.146.43.225:4457 , piece_id=JOLWRWLFRDI7QKKX7W5J3HF3KTUTDGAAFAGA4QUQEE3BYMKSHULQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.349671437 (190h9m28.780751568s) [noise=true , address=45.91.116.13:4156 , piece_id=R5LV74K2G4FDMMDN5KGXKVE3OMMJOI7KADTKMMGR65HQKSN75H7A, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.349896408 (190h9m28.780976539s) [noise=true , address=104.36.82.60:4363 , piece_id=LXSSPEY5KIBWFGF72KNGJT3O2APIGQI3OSO6V2LYK44AJKJS4EUA, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.350263870 (190h9m28.781344002s) [noise=true , address=77.90.31.21:4904 , piece_id=3Y23WQVY24KO5OIG7AAN3IFEVJSA6YYD32LTHR7JYBMUNJLGETUQ, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:34.350605874 (190h9m28.781686005s) [noise=true , address=104.36.82.50:4046 , piece_id=ZXQ67UMMM6ELR2SQNVCG77JQMGB7MPFHJRNISTMO6D4WAQCH336Q, seg_pos={0 0}, upload=197838]: Uploading piece... Aug 16 15:35:35.499261601 (190h9m29.930341732s) [noise=true , address=156.146.43.225:4457 , piece_id=JOLWRWLFRDI7QKKX7W5J3HF3KTUTDGAAFAGA4QUQEE3BYMKSHULQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:12v4SYj7tY4VFY9LqBpQDrpSmAbFEtCpPTcasMwtcSUtiQWMnJT, address:156.146.43.225:4457): protocol: storage node overloaded, request limit: 2000 Aug 16 15:35:35.816379056 (190h9m30.247459187s) [upload=197838]: Next returned nil segment Aug 16 15:35:35.816398874 (190h9m30.247478995s) [upload=197838]: Waiting for error group managing segments... Aug 16 15:35:37.436650315 (190h9m31.867730466s) [noise=true , address=79.127.201.214:4319 , piece_id=CNVVJMCF7JZNNENKZNE43U6Q6F4KIP5C3XPWLX7CLFW2G7BON2GQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:38.460355091 (190h9m32.891435212s) [noise=true , address=79.127.220.101:4061 , piece_id=AG5IA6YX742WBIJWBAAZVKGZOXLFP5IFM33NY7S6TQLSPR6JMIQA, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:40.185907245 (190h9m34.616987376s) [noise=true , address=79.127.201.215:4345 , piece_id=IJDJ7LKF7HYN6CQWTWQU2NYWADCTSB7CK3DIV2HONYGZKUSRQRCQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:41.924608920 (190h9m36.355689051s) [noise=true , address=104.36.82.50:4046 , piece_id=ZXQ67UMMM6ELR2SQNVCG77JQMGB7MPFHJRNISTMO6D4WAQCH336Q, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:49.600569822 (190h9m44.031649943s) [noise=true , address=77.90.31.21:4904 , piece_id=3Y23WQVY24KO5OIG7AAN3IFEVJSA6YYD32LTHR7JYBMUNJLGETUQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:53.836021769 (190h9m48.2671019s) [noise=true , address=104.36.82.5:4848 , piece_id=SUWUAU66F47ONBE5EZGCYHCK4KH2ZATW25DO2GD2YPJ2SYNBPHYA, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:35:56.549110714 (190h9m50.980190845s) [noise=true , address=45.91.116.13:4156 , piece_id=R5LV74K2G4FDMMDN5KGXKVE3OMMJOI7KADTKMMGR65HQKSN75H7A, seg_pos={0 0}, upload=197838]: Done uploading piece. err: <nil> Aug 16 15:36:33.244990369 (190h10m27.6760705s) [noise=true , address=104.36.82.9:4025 , piece_id=YQFBM7BQH3EB7PPBTZRCPMWCRZEQUG5N2IX6WYI6URFJSYOG5U4A, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: rpc: dial tcp 104.36.82.9:4025: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245032601 (190h10m27.676112722s) [noise=true , address=104.36.82.39:4759 , piece_id=GTAZTKTF4W2ILKHDXLPYNYFNRDZZIRIXDDSXTJWYEEPDNAWO764A, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:12AEfTC9phpBk8uTbN6k7oDvz6svgwdPH1PaU8jfuuBxhHTRtTR, address:104.36.82.39:4759): protocol: rpc: dial tcp 104.36.82.39:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245069561 (190h10m27.676149692s) [noise=true , address=104.36.82.4:4816 , piece_id=VSCXL6AXY5AXOVDMZNFGSIRZ72CR2MINZLJMAXNXZWSA45UXURIQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:12mJkBoAJW49vAg52Ts4EgdmC6BoknUngs6ZqC136rGzdSdVLUy, address:104.36.82.4:4816): protocol: write tcp 188.166.225.170:43360->104.36.82.4:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245361820 (190h10m27.676441952s) [noise=true , address=104.36.82.59:4333 , piece_id=D5APYDNIGSAPDJ7JWVJO4J2LN3GHZWGTFFZQP5EPKBFV6JP2OEZQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN, address:104.36.82.59:4333): protocol: rpc: dial tcp 104.36.82.59:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245412247 (190h10m27.676492368s) [noise=true , address=104.36.82.35:4585 , piece_id=5K2GZ7HQHPFOEEEXNGSDNGFW77IH3FKSJLMQLWIGOEVDAUOTKBBQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:104.36.82.35:4585): protocol: rpc: dial tcp 104.36.82.35:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245448336 (190h10m27.676528467s) [noise=true , address=104.36.82.36:4639 , piece_id=Z7VRH7LG2VYSBLZNDBPCYBE5YASQTR7GFLVSZKG26SV2TRRN3R3A, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:1Vs7b4tkgb1GE6i5Kswt2kQ81Gyz7kf1bf8VyzrWxCXgbJdcey, address:104.36.82.36:4639): protocol: rpc: dial tcp 104.36.82.36:4639: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245472893 (190h10m27.676553014s) [noise=true , address=77.90.31.13:4632 , piece_id=ZN2MD6ROTXBYLIF7JGURINOFO35WWEWHDG36ONNOKEQJEQPLMRTA, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245493623 (190h10m27.676573754s) [noise=true , address=104.36.82.58:4290 , piece_id=MXU4EOS5XQFFHBQMVCUYTVPH4P4Z24G62A26AZ5SSKNFG4NHZYWQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:1YieSs1jRT5hKVKc7DuME4rFeX2BmeNngjBaGoB2hRrNwiLakv, address:104.36.82.58:4290): protocol: rpc: dial tcp 104.36.82.58:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245508200 (190h10m27.676588331s) [noise=true , address=104.36.82.18:4383 , piece_id=MKDDFLJ6QLR7HXL2QMZ3IFRQPG7J6OWGTC3KLF63IC7IFRJ3OZXQ, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z, address:104.36.82.18:4383): protocol: rpc: dial tcp 104.36.82.18:4383: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245532717 (190h10m27.676612848s) [noise=true , address=104.36.82.60:4363 , piece_id=LXSSPEY5KIBWFGF72KNGJT3O2APIGQI3OSO6V2LYK44AJKJS4EUA, seg_pos={0 0}, upload=197838]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 188.166.225.170:59980->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:33.245555962 (190h10m27.676636093s) [seg_pos={0 0}, upload=197838]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:36:33.476440328 (190h10m27.907520459s) [upload=197838]: Done uploading segments... Aug 16 15:36:33.476466488 (190h10m27.907546619s) [upload=197838]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.232996868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................X......X..XXXXXX] duration: 30.229351783s FAIL (2025-08-16 15:36:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:35:39.052702894 (190h9m33.483783025s) [upload=197840]: Starting upload Aug 16 15:35:39.052758700 (190h9m33.483838821s) [upload=197840]: Done starting upload Aug 16 15:35:39.052873760 (190h9m33.483953891s) [upload=197840]: Uploading segments... Aug 16 15:35:39.052964965 (190h9m33.484045096s) [upload=197840]: Got next segment. Inline: false Aug 16 15:35:39.052997086 (190h9m33.484077217s) [upload=197840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:35:39.053001835 (190h9m33.484081966s) [upload=197840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:35:39.290926532 (190h9m33.722006673s) [seg_pos={0 0}, upload=197840]: Begin upload segment... Aug 16 15:35:39.290974484 (190h9m33.722054645s) [seg_pos={0 0}, upload=197840]: Done begin upload segment. Aug 16 15:35:39.291025822 (190h9m33.722105963s) [noise=true , address=45.91.116.11:4117 , piece_id=APVOVPI55RJKSNDCN7ZKWCIXNM5P4CN6UTI262SMEOX4FZLKHIEA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.291151292 (190h9m33.722231433s) [noise=true , address=77.90.31.18:4805 , piece_id=DV3EF7MS2QS4ZZ5KANNOUVRTZOYMKUD6DFRIALK4GIVXSV3U5D5Q, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.291483557 (190h9m33.722563699s) [noise=true , address=104.36.82.10:4079 , piece_id=4V6IAXDSIT7WH3TPFSC7XEANTFEINSASMUUELRXBLOX52N5YUPCQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.291849248 (190h9m33.722929379s) [noise=true , address=104.36.82.12:4149 , piece_id=R5X7P4GZKP7TEO4J7FUTF7IJQPRL7OK5YVAEEVRPQFDJJQLUWKBA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.291877372 (190h9m33.722957513s) [noise=true , address=104.36.82.6:4876 , piece_id=J6UT3PXJDBZRSQJYWRXAYZH2SUSYCQLE2QVBZ6AFYJL4EYGR47DQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.292250665 (190h9m33.723330806s) [noise=true , address=104.36.82.7:4909 , piece_id=UR6IHEFWKKBZJB33FGN2FA3UO34U2XPRNXN7XZ4HZVCU5MDCS5QA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.292625874 (190h9m33.723706055s) [noise=true , address=77.90.31.21:4879 , piece_id=VF7KCY6O4NZQDE7MX6FDDKAXYB2GAY4QMEXPRY72YNMCAMEESVEQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.292733479 (190h9m33.723813611s) [noise=true , address=104.36.82.8:4969 , piece_id=M4WZO6OY6Y464AXDTK7BE5GR6NDNSMDYQ75DEGVDQF5TX7RYMHAA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.292968580 (190h9m33.724048711s) [noise=true , address=45.91.116.4:4940 , piece_id=NLGXD6IRZWIHCYBWVABVQQFTIKIWKHWBWGLCOZ3DODBHDY5OBGFQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.293282740 (190h9m33.724362892s) [noise=true , address=104.36.82.6:4873 , piece_id=EEG5EXJ4TLN4J6W6HMCGOLHFOSEN3CK7KBSURJ64ENF33X6UEIMQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.293626769 (190h9m33.72470689s) [noise=true , address=104.36.82.5:4839 , piece_id=AJIMW24O7ELF2NWM7VDHYQFEXD7UKPFNJ7BFUN42MD4VBOJZ7DTQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.293931212 (190h9m33.725011353s) [noise=true , address=104.36.82.15:4258 , piece_id=E3B53UCF2R2R7ZSYRH2LBIGJCV5VH7WCLZEENA7UBTKJIOTUSM3A, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.294012177 (190h9m33.725092298s) [noise=true , address=45.91.116.12:4225 , piece_id=KWBZYXI5XXQ4GN6O53IPQ6YKLD77VGI6IPCGF6L746EHJ44ZQBNA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.294254591 (190h9m33.725334722s) [noise=true , address=77.90.31.16:4733 , piece_id=5DX37UG25EZXZVP44OWIVFRGSAUFEOCWIO5AVU7APXJ4ALWWO2JA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.294412352 (190h9m33.725492483s) [noise=true , address=77.90.31.10:4519 , piece_id=GTCX5RQO52QMJ4PVTUVW4PAXOMXOGGB3G4RPKQ2IBC7WLXWVBMUA, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:39.294964799 (190h9m33.72604494s) [noise=true , address=104.36.82.9:4007 , piece_id=EH5IDCXBNLPAXVG6ER5ZK3LWKE2HM6A7QYWUDBIEMAX2O5DVU2IQ, seg_pos={0 0}, upload=197840]: Uploading piece... Aug 16 15:35:41.063323370 (190h9m35.494403491s) [upload=197840]: Next returned nil segment Aug 16 15:35:41.063366291 (190h9m35.494446422s) [upload=197840]: Waiting for error group managing segments... Aug 16 15:35:41.365770955 (190h9m35.796851086s) [noise=true , address=104.36.82.12:4149 , piece_id=R5X7P4GZKP7TEO4J7FUTF7IJQPRL7OK5YVAEEVRPQFDJJQLUWKBA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:41.410018980 (190h9m35.841099111s) [noise=true , address=104.36.82.10:4079 , piece_id=4V6IAXDSIT7WH3TPFSC7XEANTFEINSASMUUELRXBLOX52N5YUPCQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:41.412336645 (190h9m35.843416786s) [noise=true , address=104.36.82.7:4909 , piece_id=UR6IHEFWKKBZJB33FGN2FA3UO34U2XPRNXN7XZ4HZVCU5MDCS5QA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:41.458791261 (190h9m35.889871392s) [noise=true , address=104.36.82.6:4876 , piece_id=J6UT3PXJDBZRSQJYWRXAYZH2SUSYCQLE2QVBZ6AFYJL4EYGR47DQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:41.475594061 (190h9m35.906674193s) [noise=true , address=104.36.82.6:4873 , piece_id=EEG5EXJ4TLN4J6W6HMCGOLHFOSEN3CK7KBSURJ64ENF33X6UEIMQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:43.594572923 (190h9m38.025653054s) [noise=true , address=104.36.82.15:4258 , piece_id=E3B53UCF2R2R7ZSYRH2LBIGJCV5VH7WCLZEENA7UBTKJIOTUSM3A, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:44.667266492 (190h9m39.098346623s) [noise=true , address=77.90.31.10:4519 , piece_id=GTCX5RQO52QMJ4PVTUVW4PAXOMXOGGB3G4RPKQ2IBC7WLXWVBMUA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:44.892941140 (190h9m39.324021271s) [noise=true , address=45.91.116.4:4940 , piece_id=NLGXD6IRZWIHCYBWVABVQQFTIKIWKHWBWGLCOZ3DODBHDY5OBGFQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:45.882544364 (190h9m40.313624495s) [noise=true , address=77.90.31.21:4879 , piece_id=VF7KCY6O4NZQDE7MX6FDDKAXYB2GAY4QMEXPRY72YNMCAMEESVEQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:46.171488091 (190h9m40.602568222s) [noise=true , address=45.91.116.12:4225 , piece_id=KWBZYXI5XXQ4GN6O53IPQ6YKLD77VGI6IPCGF6L746EHJ44ZQBNA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:35:55.270578601 (190h9m49.701658732s) [noise=true , address=104.36.82.5:4839 , piece_id=AJIMW24O7ELF2NWM7VDHYQFEXD7UKPFNJ7BFUN42MD4VBOJZ7DTQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:36:04.799223552 (190h9m59.230303683s) [noise=true , address=45.91.116.11:4117 , piece_id=APVOVPI55RJKSNDCN7ZKWCIXNM5P4CN6UTI262SMEOX4FZLKHIEA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: <nil> Aug 16 15:36:08.395580339 (190h10m2.82666047s) [noise=true , address=77.90.31.18:4805 , piece_id=DV3EF7MS2QS4ZZ5KANNOUVRTZOYMKUD6DFRIALK4GIVXSV3U5D5Q, seg_pos={0 0}, upload=197840]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:36:08.395941831 (190h10m2.827022012s) [noise=true , address=77.90.31.16:4733 , piece_id=5DX37UG25EZXZVP44OWIVFRGSAUFEOCWIO5AVU7APXJ4ALWWO2JA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:36:08.396024659 (190h10m2.82710478s) [noise=true , address=104.36.82.8:4969 , piece_id=M4WZO6OY6Y464AXDTK7BE5GR6NDNSMDYQ75DEGVDQF5TX7RYMHAA, seg_pos={0 0}, upload=197840]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:08.396058153 (190h10m2.827138284s) [noise=true , address=104.36.82.9:4007 , piece_id=EH5IDCXBNLPAXVG6ER5ZK3LWKE2HM6A7QYWUDBIEMAX2O5DVU2IQ, seg_pos={0 0}, upload=197840]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:36:08.396071640 (190h10m2.827151771s) [seg_pos={0 0}, upload=197840]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:36:08.622827425 (190h10m3.053907557s) [upload=197840]: Done uploading segments... Aug 16 15:36:08.622855639 (190h10m3.053935771s) [upload=197840]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.229367022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........X...X...X..................X.X..X.....X..........] duration: 3.024663815s FAIL (2025-08-16 15:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 19732f018bb804756b22dba19359f0a49a454ac3e7eb2ed1821a89357ff35551 != before dbdfd3204d0d81d2187b688175a686592fd67dface208267c3d97b7be3dbbb13 ======LOGS====== Aug 16 15:27:59.575363465 (190h1m54.006443596s) [upload=197718]: Starting upload Aug 16 15:27:59.575410946 (190h1m54.006491077s) [upload=197718]: Done starting upload Aug 16 15:27:59.575473425 (190h1m54.006553556s) [upload=197718]: Uploading segments... Aug 16 15:27:59.575513933 (190h1m54.006594054s) [upload=197718]: Got next segment. Inline: true Aug 16 15:27:59.575517490 (190h1m54.006597621s) [upload=197718]: Waiting for error group managing segments... Aug 16 15:27:59.575554039 (190h1m54.00663417s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:27:59.575558277 (190h1m54.006638408s) [upload=197718]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:27:59.575561062 (190h1m54.006641193s) [upload=197718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:26.369889366 (190h2m20.800969487s) [upload=197718]: Done uploading segments... Aug 16 15:28:26.369919393 (190h2m20.800999524s) [upload=197718]: Upload finished. err: <nil> Duration:* 3.024689073s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................................X......X..XXXXX] duration: 30.233010826s FAIL (2025-08-16 15:35:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 16 15:35:08.581587657 (190h9m3.012667788s) [upload=197833]: Starting upload Aug 16 15:35:08.581643865 (190h9m3.012723996s) [upload=197833]: Done starting upload Aug 16 15:35:08.581730230 (190h9m3.012810361s) [upload=197833]: Uploading segments... Aug 16 15:35:08.581852414 (190h9m3.012932545s) [upload=197833]: Got next segment. Inline: false Aug 16 15:35:08.581887030 (190h9m3.012967161s) [upload=197833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:35:08.581891258 (190h9m3.012971389s) [upload=197833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:35:08.827545976 (190h9m3.258626147s) [seg_pos={0 0}, upload=197833]: Begin upload segment... Aug 16 15:35:08.827614286 (190h9m3.258694437s) [seg_pos={0 0}, upload=197833]: Done begin upload segment. Aug 16 15:35:08.827647429 (190h9m3.258727651s) [noise=true , address=45.91.116.4:4940 , piece_id=P674CZP3URKIZTGGEZY6RV7BLQQCJDDYD3MYB4ZDLDBR6QXAZCLA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.827664462 (190h9m3.258744594s) [noise=true , address=77.90.31.12:4592 , piece_id=3OYZLPJ7AH7UIRJ566LSHAMILDGAJDU3STKDTV66EVNPKCD6KXTA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.827727523 (190h9m3.258807654s) [noise=true , address=104.36.82.4:4804 , piece_id=XHKJOMYS2ENMIDB5OR3CT7FEOAY77LJVKKCPECC5J4B5TD2COZDA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.828561579 (190h9m3.25964171s) [noise=true , address=104.36.82.15:4259 , piece_id=L44OQJLMFA4G2YM4SBUZ4HUDC7ATVXQ7WYWPQWC2IFY6HSUOCMYA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.829128984 (190h9m3.260209135s) [noise=true , address=104.36.82.10:4077 , piece_id=4XXFOTEODDV4VRKLP6CK2GSMLNU5D7XK4M6AWAUNJCE6CLQQVCAQ, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.829666824 (190h9m3.260746965s) [noise=true , address=104.36.82.8:4970 , piece_id=3IXAR6AQZ223V7NOBHQST2TBVLM4CIPXGHBH6JDWWGBRUPYW3WBA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.830169706 (190h9m3.261249857s) [noise=true , address=104.36.82.14:4223 , piece_id=WMOAJEJFSX3PFB52FBSVWF45GHXPA2MPSIKHX2EPLK4A5FGU7M6A, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.830620479 (190h9m3.26170062s) [noise=true , address=104.36.82.14:4221 , piece_id=7BAQQ7XFQ5SRL7VPGSEMP5IIXDGGFRDFHW6EAFC6OU2NPDSMI3VA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.831054760 (190h9m3.262134891s) [noise=true , address=77.90.31.13:4628 , piece_id=XCJM4MVXT5VQJSUU4HMTHZDNGZMK7FPU7QSOJR5XTR433UNQXUEA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.831481616 (190h9m3.262561747s) [noise=true , address=45.91.116.14:4191 , piece_id=LJFXXKGTG7SULMPMH3QHY5E6WCMD2HQ333RBJOLSD4MVTZMMDGOA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.831867935 (190h9m3.262948076s) [noise=true , address=104.36.82.10:4078 , piece_id=BYQBZANBQAMVYZCVOZZL6R5CGIZULMLXJFIJB5J5BZL76XGXKASA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.832222083 (190h9m3.263302214s) [noise=true , address=104.36.82.15:4257 , piece_id=CWHYIVJFLGSCAWHSMDLEFZJKR3GGEFQSYT6COHTAMBY55J52ISQA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.832566663 (190h9m3.263646794s) [noise=true , address=45.91.116.12:4226 , piece_id=FPQYNBGY46XTGMPHT675DUTEW6NP2QFYSLUYE5YRE7ICGVC33YDA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.832922453 (190h9m3.264002594s) [noise=true , address=104.36.82.12:4151 , piece_id=SCJM7NQ5WE2XIQAU7KKIEKCWW2E3N2YPZHWSZXTAJUAQAA2STVGQ, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.833285728 (190h9m3.264365859s) [noise=true , address=104.36.82.16:4293 , piece_id=USKCHX46V6267QP6BFWI57OS4JOZKWOHCISOQQYQ55KPNJUZJ6OQ, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:08.833591894 (190h9m3.264672025s) [noise=true , address=104.36.82.16:4295 , piece_id=3WDDVD3CBLKHM5G5DQEIGUGS2LT3ZQZWKREOYFKDIKH2XVCUKDPA, seg_pos={0 0}, upload=197833]: Uploading piece... Aug 16 15:35:10.593455901 (190h9m5.024536032s) [upload=197833]: Next returned nil segment Aug 16 15:35:10.593468054 (190h9m5.024548185s) [upload=197833]: Waiting for error group managing segments... Aug 16 15:35:10.878351513 (190h9m5.309431634s) [noise=true , address=104.36.82.12:4151 , piece_id=SCJM7NQ5WE2XIQAU7KKIEKCWW2E3N2YPZHWSZXTAJUAQAA2STVGQ, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:10.878934548 (190h9m5.310014679s) [noise=true , address=104.36.82.15:4257 , piece_id=CWHYIVJFLGSCAWHSMDLEFZJKR3GGEFQSYT6COHTAMBY55J52ISQA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:10.949040249 (190h9m5.38012037s) [noise=true , address=104.36.82.15:4259 , piece_id=L44OQJLMFA4G2YM4SBUZ4HUDC7ATVXQ7WYWPQWC2IFY6HSUOCMYA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:10.966493003 (190h9m5.397573134s) [noise=true , address=104.36.82.10:4077 , piece_id=4XXFOTEODDV4VRKLP6CK2GSMLNU5D7XK4M6AWAUNJCE6CLQQVCAQ, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:10.989512589 (190h9m5.420592721s) [noise=true , address=104.36.82.10:4078 , piece_id=BYQBZANBQAMVYZCVOZZL6R5CGIZULMLXJFIJB5J5BZL76XGXKASA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:11.123139412 (190h9m5.554219543s) [noise=true , address=45.91.116.12:4226 , piece_id=FPQYNBGY46XTGMPHT675DUTEW6NP2QFYSLUYE5YRE7ICGVC33YDA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:11.399352981 (190h9m5.830433112s) [noise=true , address=77.90.31.12:4592 , piece_id=3OYZLPJ7AH7UIRJ566LSHAMILDGAJDU3STKDTV66EVNPKCD6KXTA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:11.579379585 (190h9m6.010459716s) [noise=true , address=45.91.116.4:4940 , piece_id=P674CZP3URKIZTGGEZY6RV7BLQQCJDDYD3MYB4ZDLDBR6QXAZCLA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:12.005169404 (190h9m6.436249535s) [noise=true , address=45.91.116.14:4191 , piece_id=LJFXXKGTG7SULMPMH3QHY5E6WCMD2HQ333RBJOLSD4MVTZMMDGOA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: <nil> Aug 16 15:35:37.931906543 (190h9m32.362986694s) [noise=true , address=104.36.82.14:4221 , piece_id=7BAQQ7XFQ5SRL7VPGSEMP5IIXDGGFRDFHW6EAFC6OU2NPDSMI3VA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:47882->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.931929316 (190h9m32.363009447s) [noise=true , address=104.36.82.14:4223 , piece_id=WMOAJEJFSX3PFB52FBSVWF45GHXPA2MPSIKHX2EPLK4A5FGU7M6A, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.931964354 (190h9m32.363044485s) [noise=true , address=104.36.82.8:4970 , piece_id=3IXAR6AQZ223V7NOBHQST2TBVLM4CIPXGHBH6JDWWGBRUPYW3WBA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.931994671 (190h9m32.363074813s) [noise=true , address=77.90.31.13:4628 , piece_id=XCJM4MVXT5VQJSUU4HMTHZDNGZMK7FPU7QSOJR5XTR433UNQXUEA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:52226->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.932009119 (190h9m32.36308927s) [noise=true , address=104.36.82.4:4804 , piece_id=XHKJOMYS2ENMIDB5OR3CT7FEOAY77LJVKKCPECC5J4B5TD2COZDA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.932026783 (190h9m32.363106914s) [noise=true , address=104.36.82.16:4293 , piece_id=USKCHX46V6267QP6BFWI57OS4JOZKWOHCISOQQYQ55KPNJUZJ6OQ, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.932045990 (190h9m32.363126121s) [noise=true , address=104.36.82.16:4295 , piece_id=3WDDVD3CBLKHM5G5DQEIGUGS2LT3ZQZWKREOYFKDIKH2XVCUKDPA, seg_pos={0 0}, upload=197833]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:43802->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.932058032 (190h9m32.363138153s) [seg_pos={0 0}, upload=197833]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 16 15:35:38.162885710 (190h9m32.593965841s) [upload=197833]: Done uploading segments... Aug 16 15:35:38.162911179 (190h9m32.59399131s) [upload=197833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.233031334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X...........X......X...X.X......X..X......X..X...XXXXXXX] duration: 1m0.337662997s FAIL (2025-08-16 15:35:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:34:38.331282110 (190h8m32.762362242s) [upload=197824]: Starting upload Aug 16 15:34:38.331341634 (190h8m32.762421765s) [upload=197824]: Done starting upload Aug 16 15:34:38.331444611 (190h8m32.762524743s) [upload=197824]: Uploading segments... Aug 16 15:34:38.331558910 (190h8m32.762639041s) [upload=197824]: Got next segment. Inline: false Aug 16 15:34:38.331593005 (190h8m32.762673136s) [upload=197824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:38.331597083 (190h8m32.762677214s) [upload=197824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:38.840954622 (190h8m33.272034753s) [seg_pos={0 0}, upload=197824]: Begin upload segment... Aug 16 15:34:38.841013494 (190h8m33.272093615s) [seg_pos={0 0}, upload=197824]: Done begin upload segment. Aug 16 15:34:38.841035887 (190h8m33.272116008s) [noise=true , address=104.36.82.38:4711 , piece_id=6EFKGHODXCMEDZDS6GCPTTQ22E6V2GUUOP73VK6CRZNLTEYIEECA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.841092926 (190h8m33.272173057s) [noise=true , address=45.91.116.12:4239 , piece_id=OLC7TFG4NEMPC4NRA6XZUP4W7SEFOMYCEUETZK24IECGQVM6VIDQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.841465659 (190h8m33.27254579s) [noise=true , address=104.36.82.2:4744 , piece_id=I62NZAVYSYBSCCRH6HEKPINOSTZEI6FZ2Q3VMFRVDVRIMFOFOIWQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.841637298 (190h8m33.272717429s) [noise=true , address=104.36.82.59:4303 , piece_id=4NVAX7G6MROSDZSJVYNEQXON4M7GLLVFWHMESTWIM2RMHN5LELYQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.841769079 (190h8m33.27284921s) [noise=true , address=77.90.31.21:4892 , piece_id=XINIYMBSDKXNIZPX2QZIIXJ3BTCFOBGUNGF7WNRI6WHUEY65DUYA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842003729 (190h8m33.27308386s) [noise=true , address=104.36.82.39:4730 , piece_id=324JRLIJGVYUZRCINSKJKRHPKCBQAMUV4ZOIYNMKRVO6TRLWKNXQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842093711 (190h8m33.273173832s) [noise=true , address=77.90.31.18:4823 , piece_id=BR37MUQIF2S2O7JDI4LXRAZ2VQVLES4WH2MK3MMCDEIDPQJEIUZQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842366172 (190h8m33.273446303s) [noise=true , address=66.23.193.175:4558 , piece_id=KP7YHSZ7OKWMQJ35BL2CNTB3XLP4DJZTSRVBKHN4RWJQFOTRQKZQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842370520 (190h8m33.273450651s) [noise=true , address=45.91.116.5:4324 , piece_id=XZP4FQJWBA4RXQRCJXIZZIRCJYWFCTZEG4ODE4BEHQVW2EPM4YWA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842645356 (190h8m33.273725487s) [noise=true , address=104.36.82.19:4424 , piece_id=YG67TQCCY3VDBRAEKPNTSDIIRFKHQB4ICHI5UQMGZFTA5QSMJFCQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842729066 (190h8m33.273809197s) [noise=true , address=104.36.82.44:4872 , piece_id=QATBX2BIXF7ZZ4Z3YE4X2MJRFII6CO5R5TQCKC72X66HGH6AQ5WA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842846061 (190h8m33.273926192s) [noise=true , address=77.90.31.15:4700 , piece_id=NPWIQZSOJQ7D3DVJFHJGJYKDX37DL5YI6E2YHDT7MTIMYQJI4EWQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.842989946 (190h8m33.274070077s) [noise=true , address=79.127.201.215:4349 , piece_id=3DHKMIWINESZ6DQN72X6YVFT2ZDKMBPZAJXQEQUP2MEMN6CZVDJA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.843052756 (190h8m33.274132877s) [noise=true , address=104.36.82.60:4364 , piece_id=UFBP7FV6R4EWZ5KXU4IS7I4HPVOMYYYEENSHSZYSSQVTPOPYJ2DQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.843254062 (190h8m33.274334193s) [noise=true , address=77.90.31.12:4595 , piece_id=YBCMT7GE6Q4AIFIJW6ZNJPXR4BHTQ4BTEL5XZKIS5JMHTINDHJBA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.843289459 (190h8m33.27436959s) [noise=true , address=104.36.82.5:4860 , piece_id=OLX6VVQGHOPUCUDWSGXUVLFXLVKQJI236U7PSFBBU2X5HE4B2FXA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.843491425 (190h8m33.274571546s) [noise=true , address=45.91.116.6:4985 , piece_id=YSQ66QAM6KTM367JZVFZZBG5MXLHIMZLQHBKWEUJAHH7SHSE5KSA, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:38.844366790 (190h8m33.275446921s) [noise=true , address=104.36.82.37:4687 , piece_id=HVZ3KDTYF7NOORFGUSUEFP2T2JLUTS3UFNPV2IGHKEZ7LVJNTVSQ, seg_pos={0 0}, upload=197824]: Uploading piece... Aug 16 15:34:40.086560844 (190h8m34.517640975s) [upload=197824]: Next returned nil segment Aug 16 15:34:40.086574611 (190h8m34.517654732s) [upload=197824]: Waiting for error group managing segments... Aug 16 15:34:40.958688260 (190h8m35.389768381s) [noise=true , address=77.90.31.12:4595 , piece_id=YBCMT7GE6Q4AIFIJW6ZNJPXR4BHTQ4BTEL5XZKIS5JMHTINDHJBA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:41.139450491 (190h8m35.570530622s) [noise=true , address=77.90.31.21:4892 , piece_id=XINIYMBSDKXNIZPX2QZIIXJ3BTCFOBGUNGF7WNRI6WHUEY65DUYA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:41.167734721 (190h8m35.598814852s) [noise=true , address=77.90.31.15:4700 , piece_id=NPWIQZSOJQ7D3DVJFHJGJYKDX37DL5YI6E2YHDT7MTIMYQJI4EWQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:41.198112706 (190h8m35.629192837s) [noise=true , address=45.91.116.6:4985 , piece_id=YSQ66QAM6KTM367JZVFZZBG5MXLHIMZLQHBKWEUJAHH7SHSE5KSA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:41.597604597 (190h8m36.028684728s) [noise=true , address=45.91.116.5:4324 , piece_id=XZP4FQJWBA4RXQRCJXIZZIRCJYWFCTZEG4ODE4BEHQVW2EPM4YWA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:43.469763009 (190h8m37.90084314s) [noise=true , address=77.90.31.18:4823 , piece_id=BR37MUQIF2S2O7JDI4LXRAZ2VQVLES4WH2MK3MMCDEIDPQJEIUZQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:43.495653077 (190h8m37.926733198s) [noise=true , address=45.91.116.12:4239 , piece_id=OLC7TFG4NEMPC4NRA6XZUP4W7SEFOMYCEUETZK24IECGQVM6VIDQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:34:46.890081994 (190h8m41.321162125s) [noise=true , address=66.23.193.175:4558 , piece_id=KP7YHSZ7OKWMQJ35BL2CNTB3XLP4DJZTSRVBKHN4RWJQFOTRQKZQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:35:07.741582785 (190h9m2.172662916s) [noise=true , address=79.127.201.215:4349 , piece_id=3DHKMIWINESZ6DQN72X6YVFT2ZDKMBPZAJXQEQUP2MEMN6CZVDJA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: <nil> Aug 16 15:35:37.678289419 (190h9m32.10936955s) [noise=true , address=104.36.82.59:4303 , piece_id=4NVAX7G6MROSDZSJVYNEQXON4M7GLLVFWHMESTWIM2RMHN5LELYQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, address:104.36.82.59:4303): protocol: rpc: dial tcp 104.36.82.59:4303: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678333494 (190h9m32.109413625s) [noise=true , address=104.36.82.5:4860 , piece_id=OLX6VVQGHOPUCUDWSGXUVLFXLVKQJI236U7PSFBBU2X5HE4B2FXA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678362730 (190h9m32.109442861s) [noise=true , address=104.36.82.2:4744 , piece_id=I62NZAVYSYBSCCRH6HEKPINOSTZEI6FZ2Q3VMFRVDVRIMFOFOIWQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: rpc: dial tcp 104.36.82.2:4744: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678391825 (190h9m32.109471956s) [noise=true , address=104.36.82.37:4687 , piece_id=HVZ3KDTYF7NOORFGUSUEFP2T2JLUTS3UFNPV2IGHKEZ7LVJNTVSQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12JZW19SfSwU95frntTgqDhiam1Rva1U8V7yfQkqtKYWntJ24Qf, address:104.36.82.37:4687): protocol: write tcp 188.166.225.170:39168->104.36.82.37:4687: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678423476 (190h9m32.109503607s) [noise=true , address=104.36.82.44:4872 , piece_id=QATBX2BIXF7ZZ4Z3YE4X2MJRFII6CO5R5TQCKC72X66HGH6AQ5WA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: rpc: dial tcp 104.36.82.44:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678454926 (190h9m32.109535057s) [noise=true , address=104.36.82.38:4711 , piece_id=6EFKGHODXCMEDZDS6GCPTTQ22E6V2GUUOP73VK6CRZNLTEYIEECA, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12hpgtWXaoVC18JQaeCK21b7N7B78aYGU8csoEjCv5FLkuciQmM, address:104.36.82.38:4711): protocol: rpc: dial tcp 104.36.82.38:4711: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678472420 (190h9m32.109552551s) [noise=true , address=104.36.82.19:4424 , piece_id=YG67TQCCY3VDBRAEKPNTSDIIRFKHQB4ICHI5UQMGZFTA5QSMJFCQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: rpc: dial tcp 104.36.82.19:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678489332 (190h9m32.109569463s) [noise=true , address=104.36.82.39:4730 , piece_id=324JRLIJGVYUZRCINSKJKRHPKCBQAMUV4ZOIYNMKRVO6TRLWKNXQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:104.36.82.39:4730): protocol: write tcp 188.166.225.170:44322->104.36.82.39:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678518047 (190h9m32.109598178s) [noise=true , address=104.36.82.60:4364 , piece_id=UFBP7FV6R4EWZ5KXU4IS7I4HPVOMYYYEENSHSZYSSQVTPOPYJ2DQ, seg_pos={0 0}, upload=197824]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:104.36.82.60:4364): protocol: write tcp 188.166.225.170:48152->104.36.82.60:4364: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:37.678541162 (190h9m32.109621293s) [seg_pos={0 0}, upload=197824]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:35:38.014058062 (190h9m32.445138193s) [upload=197824]: Done uploading segments... Aug 16 15:35:38.014091025 (190h9m32.445171156s) [upload=197824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.337678978s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)