Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.542814561s FAIL (2025-08-17 13:19:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.542835471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.320867147s FAIL (2025-08-17 13:19:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.320902305s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................X.......X...............X.....X.........] duration: 1.85959911s FAIL (2025-08-17 13:19:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.859620971s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration: 5.898684361s FAIL (2025-08-17 13:19:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.898708868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.69440338s FAIL (2025-08-17 13:19:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.694425471s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXXXXX.................] duration: 5.939368984s FAIL (2025-08-17 13:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.939391197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X......X..X......XXX.X...................] duration: 6.174533794s FAIL (2025-08-17 13:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.17456394s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXXXXX................] duration: 5.895760713s FAIL (2025-08-17 13:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.895804357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 5.85248336s FAIL (2025-08-17 13:19:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.852512404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.......X...............X.....X........] duration: 1.862229094s FAIL (2025-08-17 13:18:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.862257308s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.42886393s FAIL (2025-08-17 13:18:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.428879692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 6.116607978s FAIL (2025-08-17 13:18:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.116636722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X......X..X......XXX.X..................] duration: 5.959715986s FAIL (2025-08-17 13:18:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.959741526s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXXXXX...............] duration: 5.906320152s FAIL (2025-08-17 13:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.906347425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.464345688s FAIL (2025-08-17 13:18:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.464371506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................X.......X...............X.....X.......] duration: 1.868467298s FAIL (2025-08-17 13:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.86849467s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X......X..X......XXX.X.................] duration: 5.591011684s FAIL (2025-08-17 13:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.591037271s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 5.991141702s FAIL (2025-08-17 13:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.991210163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.860159807s FAIL (2025-08-17 13:17:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.860178101s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXXXXX..............] duration: 5.849458392s FAIL (2025-08-17 13:17:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.849477628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 5.929304771s FAIL (2025-08-17 13:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.929324909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.717924505s FAIL (2025-08-17 13:17:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.717949092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.095095157s FAIL (2025-08-17 13:17:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.095121618s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXXXXX.............] duration: 5.92468072s FAIL (2025-08-17 13:17:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.9247118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.514537477s FAIL (2025-08-17 13:17:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.514559459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.......X...............X.....X......] duration: 2.19668341s FAIL (2025-08-17 13:16:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.196710532s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.320254038s FAIL (2025-08-17 13:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.32027063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X......X..X......XXX.X................] duration: 5.768587685s FAIL (2025-08-17 13:16:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.768613143s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration: 5.902897562s FAIL (2025-08-17 13:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.90292321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.332911438s FAIL (2025-08-17 13:16:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.33292869s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 5.729770632s FAIL (2025-08-17 13:16:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.729794538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X......X..X......XXX.X...............] duration: 5.786430114s FAIL (2025-08-17 13:16:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.786457896s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.682366618s FAIL (2025-08-17 13:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.682412666s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXXXXX............] duration: 5.950223009s FAIL (2025-08-17 13:16:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.950244449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.466345131s FAIL (2025-08-17 13:16:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.466368847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................X.......X...............X.....X.....] duration: 2.177107331s FAIL (2025-08-17 13:16:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.177134293s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 5.878784533s FAIL (2025-08-17 13:15:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.878807007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X......X..X......XXX.X..............] duration: 5.700835149s FAIL (2025-08-17 13:15:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.700856981s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.876365437s FAIL (2025-08-17 13:15:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.876394333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.163686503s FAIL (2025-08-17 13:15:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.163714136s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XXXXXX...........] duration: 6.085735347s FAIL (2025-08-17 13:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.085756207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X......X..X......XXX.X.............] duration: 5.995398843s FAIL (2025-08-17 13:15:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.995427617s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.......X...............X.....X....] duration: 1.914445281s FAIL (2025-08-17 13:14:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:11:38.665184940 (211h45m33.096265071s) [upload=220254]: Starting upload Aug 17 13:11:38.665224617 (211h45m33.096304748s) [upload=220254]: Done starting upload Aug 17 13:11:38.665268069 (211h45m33.0963482s) [upload=220254]: Uploading segments... Aug 17 13:11:38.665298477 (211h45m33.096378639s) [upload=220254]: Got next segment. Inline: true Aug 17 13:11:38.665302525 (211h45m33.096382656s) [upload=220254]: Waiting for error group managing segments... Aug 17 13:11:38.665333465 (211h45m33.096413596s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:11:38.665337052 (211h45m33.096417183s) [upload=220254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:11:38.665339496 (211h45m33.096419627s) [upload=220254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:11:38.891348627 (211h45m33.322428758s) [upload=220254]: Done uploading segments... Aug 17 13:11:38.891361952 (211h45m33.322442083s) [upload=220254]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.914468446s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X......X..X......XXX.X............] duration: 5.986640188s FAIL (2025-08-17 13:14:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.986673421s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 5.736152364s FAIL (2025-08-17 13:14:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.736170228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.646017869s FAIL (2025-08-17 13:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.646037467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X......X..X......XXX.X...........] duration: 6.049128624s FAIL (2025-08-17 13:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 17 13:05:31.920394071 (211h39m26.351474222s) [upload=220147]: Starting upload Aug 17 13:05:31.920504732 (211h39m26.351584873s) [upload=220147]: Done starting upload Aug 17 13:05:31.920616867 (211h39m26.351696988s) [upload=220147]: Uploading segments... Aug 17 13:05:31.920943512 (211h39m26.352023693s) [upload=220147]: Got next segment. Inline: false Aug 17 13:05:31.920993747 (211h39m26.352073878s) [upload=220147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:31.920998908 (211h39m26.352079039s) [upload=220147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:32.402992083 (211h39m26.834072214s) [seg_pos={0 0}, upload=220147]: Begin upload segment... Aug 17 13:05:32.403034935 (211h39m26.834115056s) [seg_pos={0 0}, upload=220147]: Done begin upload segment. Aug 17 13:05:32.403056958 (211h39m26.834137089s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403139515 (211h39m26.834219647s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403424030 (211h39m26.834504161s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403557345 (211h39m26.834637476s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403767457 (211h39m26.834847588s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.403891855 (211h39m26.834971986s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404057051 (211h39m26.835137182s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404213851 (211h39m26.835293972s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404344812 (211h39m26.835424943s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404580733 (211h39m26.835660854s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404739396 (211h39m26.835819527s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404816023 (211h39m26.835896144s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.404960559 (211h39m26.836040691s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405041504 (211h39m26.836121635s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405259021 (211h39m26.836339152s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405292294 (211h39m26.836372425s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405489211 (211h39m26.836569342s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:32.405597438 (211h39m26.836677569s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Uploading piece... Aug 17 13:05:33.641687857 (211h39m28.072767999s) [upload=220147]: Next returned nil segment Aug 17 13:05:33.641709419 (211h39m28.07278955s) [upload=220147]: Waiting for error group managing segments... Aug 17 13:05:33.965595805 (211h39m28.396675936s) [noise=true , address=66.23.193.175:4553 , piece_id=4RN7ZF4VDYPR2JOGGIYVOLWIT3KCM7WT4ZSRZWN2UJZTBTLLTKKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.241173750 (211h39m28.672253911s) [noise=true , address=79.127.220.101:4068 , piece_id=ONEZBI7AZCDCIPYJSGT4SHQ464FW3Y64F5L6XJ5AVBPAVPOGZNYA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.248850439 (211h39m28.67993057s) [noise=true , address=79.127.220.97:4385 , piece_id=VB56X7M45BE5GQ4W6OM35DV4LXDL5MOCED3DEZOFLBTQ233K35NA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.316295727 (211h39m28.747375858s) [noise=true , address=79.127.219.47:4057 , piece_id=S3BEOTJAGJNRM4GERMDLNJAUGFMPZHSUSYOUKBURYTGRAGD6XFKA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.360061158 (211h39m28.791141279s) [noise=true , address=45.91.116.14:4220 , piece_id=3EE567WVFRDBOFCMWU5WJZQRSJPLCYLTNSUGFM35QD4FVIQNBIZQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.368159373 (211h39m28.799239504s) [noise=true , address=156.146.43.225:4456 , piece_id=6VSLO57VEHMHHHEWWAP7CNMQK76TCKL24JBRQ2XGUR3V7RLUVJMQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.384299364 (211h39m28.815379485s) [noise=true , address=45.91.116.7:4036 , piece_id=UPXTO5ZUTN3QBFIIF7BK2HKGO7BJNEJKPJEK2AC7G6OCQTQI666A, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.415796530 (211h39m28.846876661s) [noise=true , address=77.90.31.11:4556 , piece_id=5ST4LHX2JZDS4YZ3PZ2U3BLU225UTHZEXMEDRC4HHJQ5FCKY6VPA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.453896940 (211h39m28.884977061s) [noise=true , address=77.90.31.21:4880 , piece_id=76ZDGC7N5UQIQWGD7YHTIY7SRUCBF7KWLTXPDWBCF4UZXFPSWCRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.504351208 (211h39m28.935431339s) [noise=true , address=45.91.116.11:4146 , piece_id=XR2U5W6YRCPWVLT5TAMDYZAD6GDYMFLEE6GM2JVTSASVJDQ5KFAA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:05:34.781278205 (211h39m29.212358326s) [noise=true , address=77.90.31.18:4828 , piece_id=LTVXRVR76O5XLMYCN5TAVZ2MN4ZTMMFK4GJVOPYCIFWTBO5HMCKQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: <nil> Aug 17 13:06:31.265285084 (211h40m25.696365215s) [noise=true , address=104.36.82.49:4010 , piece_id=DMF5XH47X6ZDPXF64WXAT4GWSVUTEI6FLARDD5HY7L6R4VR3T4KA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:104.36.82.49:4010): protocol: rpc: dial tcp 104.36.82.49:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265329900 (211h40m25.696410032s) [noise=true , address=104.36.82.17:4361 , piece_id=5YCMCBH5WRYYZXWVZPXHDJP2BCQCMSP3I6FI6H5QWFHG7J3DZLRA, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1H34KhE8VwBXA9y9RcjKvYmuqfDPCuokR5bRFnmbAJT9dgz9gU, address:104.36.82.17:4361): protocol: rpc: dial tcp 104.36.82.17:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265363194 (211h40m25.696443325s) [noise=true , address=104.36.82.35:4594 , piece_id=G7563OGCYIK7YUW2FSTEQXZ7RT4UWNVJ4WWXHB2M4HQFD5PLDNTQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265397069 (211h40m25.6964772s) [noise=true , address=104.36.82.10:4101 , piece_id=LSILRD2RIGYR3RUWC565C36BLFCVHHO4IZYTMC5R77EN7S3MKPEQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:34438->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265445111 (211h40m25.696525242s) [noise=true , address=104.36.82.60:4358 , piece_id=CCMYRQXFBNNA7BJK2YH6IWWPMFMB4UEQI6UP5U3Y5QNZTVPMKVNQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: write tcp 188.166.225.170:37426->104.36.82.60:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265470469 (211h40m25.6965506s) [noise=true , address=104.36.82.23:4479 , piece_id=DTQMEJOVHTUM7H2G5XZZ5HOTHPUMYDJXOHUTTL5K3CSDHMC3SH3Q, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:104.36.82.23:4479): protocol: rpc: dial tcp 104.36.82.23:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265560000 (211h40m25.696640131s) [noise=true , address=104.36.82.17:4339 , piece_id=SLVELKGPYBHLWENVYWQMQYIBCTR5VXEH47UN6M76YYKHYDKBHYUQ, seg_pos={0 0}, upload=220147]: Done uploading piece. err: ecclient: upload failed (node:1BwQctDb8mxcmcvf6jvTdQqPx7xv4QMsbaVimioZxGS5vstuvk, address:104.36.82.17:4339): protocol: rpc: dial tcp 104.36.82.17:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:31.265625345 (211h40m25.696705486s) [seg_pos={0 0}, upload=220147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 17 13:06:31.502996216 (211h40m25.934076347s) [upload=220147]: Done uploading segments... Aug 17 13:06:31.503024941 (211h40m25.934105072s) [upload=220147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.049166816s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................XXXXXX..........] duration: 5.931237146s FAIL (2025-08-17 13:14:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:05:17.433539256 (211h39m11.864619387s) [upload=220142]: Starting upload Aug 17 13:05:17.433596886 (211h39m11.864677017s) [upload=220142]: Done starting upload Aug 17 13:05:17.433662783 (211h39m11.864742904s) [upload=220142]: Uploading segments... Aug 17 13:05:17.433776180 (211h39m11.864856311s) [upload=220142]: Got next segment. Inline: false Aug 17 13:05:17.433827598 (211h39m11.864907719s) [upload=220142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:17.433832447 (211h39m11.864912578s) [upload=220142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:17.675656378 (211h39m12.106736509s) [seg_pos={0 0}, upload=220142]: Begin upload segment... Aug 17 13:05:17.675703008 (211h39m12.106783139s) [seg_pos={0 0}, upload=220142]: Done begin upload segment. Aug 17 13:05:17.675726002 (211h39m12.106806133s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.675747853 (211h39m12.106827984s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676138762 (211h39m12.107218883s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676261807 (211h39m12.107341938s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676460487 (211h39m12.107540618s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676632336 (211h39m12.107712467s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676712388 (211h39m12.107792519s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.676969802 (211h39m12.108049933s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677250579 (211h39m12.1083307s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677558217 (211h39m12.108638348s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.677812653 (211h39m12.108892785s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678043957 (211h39m12.109124088s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678292302 (211h39m12.109372433s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678574141 (211h39m12.109654272s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.678822796 (211h39m12.109902928s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:17.679099606 (211h39m12.110179737s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Uploading piece... Aug 17 13:05:19.426152620 (211h39m13.857232751s) [upload=220142]: Next returned nil segment Aug 17 13:05:19.426163030 (211h39m13.857243161s) [upload=220142]: Waiting for error group managing segments... Aug 17 13:05:19.703740491 (211h39m14.134820622s) [noise=true , address=45.91.116.11:4118 , piece_id=SSYVDYM2TAV3H6E6RU3JI5BX3CTFF2PVX3RRRFKRHMAP5QMUJ26Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.757408347 (211h39m14.188488479s) [noise=true , address=45.91.116.4:4940 , piece_id=AUMV23VN45TVS6HPT3MTMG23BOHAKMH5RT6TJAYXIXPBY2GSMQ2Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.758279454 (211h39m14.189359586s) [noise=true , address=45.91.116.14:4189 , piece_id=5J22CKRDTGKZESPNSB6PBUM5PY4PBQ6MG2ULDQ26PQNDCF62JFWA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.761251431 (211h39m14.192331602s) [noise=true , address=45.91.116.12:4227 , piece_id=VQDZBRMULAZJH7WC5V7MP6SKHF43G3QBYDAUNXQKSCFFV6WCAHEA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.775537656 (211h39m14.206617777s) [noise=true , address=77.90.31.14:4662 , piece_id=WGP7GL6IO5BGNMJUYKL3V52VHJ5DCJR4ZY4MQLDMOA26DFYSXJ3A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.862319422 (211h39m14.293399553s) [noise=true , address=45.91.116.7:4009 , piece_id=QXQ2HKOFK7K26YCCFAPZYLO7TAAJMFKKC27BVAJ4WBUHFPVNR3JA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.890846586 (211h39m14.321926717s) [noise=true , address=45.91.116.12:4226 , piece_id=MZECXOMFTP4AZ3M4JWNBWUBBV3CA7TJHNYK3RPWOFC2RYTMECAUQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.896130075 (211h39m14.327210206s) [noise=true , address=77.90.31.12:4589 , piece_id=ICTOTCQ6DTO44KO2R3YSZ6D6UKC7PRYUQMAC3RWLR5BRXG3PSE5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.956338697 (211h39m14.387418828s) [noise=true , address=45.91.116.5:4298 , piece_id=QNQUAHGFUNMMZZY7S4FG2NML6ORH4IBBPGMKDFUY4MBRDQ5YPY2A, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:19.963069265 (211h39m14.394149396s) [noise=true , address=45.91.116.6:4974 , piece_id=4QFWE2Q65YWJ4YZEVGIJSDZFNDVNR5GBMPWFX3LJ2M5WPLLYJ6MA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: <nil> Aug 17 13:05:46.757230974 (211h39m41.188311105s) [noise=true , address=104.36.82.5:4839 , piece_id=KOQHTT6MBEL2AG7GDMK5TQ7OLD2HGHIV5PHUG3DHJE6GPR7JLREQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56088->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757310557 (211h39m41.188390688s) [noise=true , address=104.36.82.7:4911 , piece_id=GSMSTQQR55EKEPXBHRO6Z7VCUPB4NNUTO6YWGNGN5PZRMM2UP5SA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:43896->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757367205 (211h39m41.188447336s) [noise=true , address=104.36.82.8:4969 , piece_id=WQZCS5XQ4RRJJRJSIIEAJ7DD53GLUA4PGCLQRA4UR4HNW7RVPOOQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:50830->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757630910 (211h39m41.188711041s) [noise=true , address=104.36.82.16:4295 , piece_id=YKNJXFJ3UYRQAHG5T24VYMGAORMMZ6632DC4INVDTD5H2G77YUFQ, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:46772->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757698278 (211h39m41.188778419s) [noise=true , address=104.36.82.12:4149 , piece_id=5NQXC5JIUHYKOFCTOG3FMR2JN6O3UC52WFTFBBPM2VHPNZNMDFCA, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:40344->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:46.757973695 (211h39m41.189053826s) [noise=true , address=104.36.82.14:4222 , piece_id=RRVUYHL2LI3OU7PQNH7JAP774F7LNWCWCZZWBLRXR7XTWY75QO5Q, seg_pos={0 0}, upload=220142]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:44768->104.36.82.14:4222: use of closed network connection; context deadline exceeded Aug 17 13:05:46.758015345 (211h39m41.189095476s) [seg_pos={0 0}, upload=220142]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:47.006930334 (211h39m41.438010465s) [upload=220142]: Done uploading segments... Aug 17 13:05:47.006956404 (211h39m41.438036535s) [upload=220142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.931262645s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.22193311s FAIL (2025-08-17 13:14:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.221957115s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 5.674602802s FAIL (2025-08-17 13:13:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:06:06.331101842 (211h40m0.762181963s) [upload=220155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:06:06.869815355 (211h40m1.300895486s) [seg_pos={0 0}, upload=220155]: Begin upload segment... Aug 17 13:06:06.869856955 (211h40m1.300937086s) [seg_pos={0 0}, upload=220155]: Done begin upload segment. Aug 17 13:06:06.869876613 (211h40m1.300956744s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.869981392 (211h40m1.301061523s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870247702 (211h40m1.301327833s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870421785 (211h40m1.301502006s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870533740 (211h40m1.301613871s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870800780 (211h40m1.301880911s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.870955607 (211h40m1.302035748s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871036973 (211h40m1.302117104s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871276161 (211h40m1.302356282s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871519846 (211h40m1.302599977s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871724548 (211h40m1.302804679s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871765757 (211h40m1.302845888s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.871979896 (211h40m1.303060028s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872053938 (211h40m1.30313407s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872172705 (211h40m1.303252836s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872306181 (211h40m1.303386312s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872577690 (211h40m1.303657822s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:06.872795978 (211h40m1.303876109s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Uploading piece... Aug 17 13:06:08.359034867 (211h40m2.790114998s) [upload=220155]: Next returned nil segment Aug 17 13:06:08.359051689 (211h40m2.79013181s) [upload=220155]: Waiting for error group managing segments... Aug 17 13:06:08.764668267 (211h40m3.195748398s) [noise=true , address=79.127.219.48:4099 , piece_id=R7LIPWYZLCD4V7TPMRDNWMJP7LXTPUUUJITNWZDDFSNWZRK5FHSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.828381334 (211h40m3.259461465s) [noise=true , address=45.91.116.6:4993 , piece_id=YAN3JXHIMX6FTTOOX4BSKA5C6OXRZXSZFWTPRXAGZYTKRWZLRP6A, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.874258294 (211h40m3.305338425s) [noise=true , address=79.127.201.215:4359 , piece_id=KRFR3X6CN2P77HJ5EJQ72PGKHH7IATP7CAVEP6NPCJWSVC3CEXOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.898595911 (211h40m3.329676042s) [noise=true , address=77.90.31.12:4612 , piece_id=SA5UMCUQZDS4IRJ6ATS27GZKRSACM6M2JLV6X5U5KYY7MZGSJZCQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:08.988269381 (211h40m3.419349512s) [noise=true , address=77.90.31.11:4583 , piece_id=I343NHYWITY3U4DXEBWBACFIV5G24L44LY37TKWXFDLOVLWNOBFA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.002563361 (211h40m3.433643492s) [noise=true , address=77.90.31.13:4636 , piece_id=6NH42MT24G4IS3WRCQI7CRA52QJI7CLTQHAK7QN257FUYX5GFVAA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.069752299 (211h40m3.50083243s) [noise=true , address=45.91.116.8:4065 , piece_id=VH57G3KTGMCA3D5HSYHNWVL7PQ5VU54Z7ES2OKRJLVHPBNSCAEGQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.089606457 (211h40m3.520686578s) [noise=true , address=77.90.31.19:4870 , piece_id=RJZIGTVUFMK7COBMWEOBOXIJ7HHSQXJD6NDISWUEVGB2V4EI7NOA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.102728826 (211h40m3.533808957s) [noise=true , address=45.91.116.11:4145 , piece_id=6YP5RITCAW55JRJ3OITTELFHBN7CZXILXCSXF53FGU4WUUAMXPJA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.137437406 (211h40m3.568517537s) [noise=true , address=77.90.31.21:4898 , piece_id=OIXWM26W5JQ4DDZ3ITAGS3N5B3BFRZRJ2SNS6S67QBAE2RJY7XSQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:09.251254211 (211h40m3.682334342s) [noise=true , address=45.91.116.12:4228 , piece_id=GGVBYIVGWLYKVPMMO4BFJNGDRY3M72EFJLXTJ3ZHP5MAUALNFRMA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:06:10.171522121 (211h40m4.602602252s) [noise=true , address=45.91.116.10:4111 , piece_id=YPXEPDOILTWPN2WNAZCZLJ22CIV72U2JBP23RKJDPIE5LDDWI3BQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: <nil> Aug 17 13:07:05.671508065 (211h41m0.102588196s) [noise=true , address=104.36.82.5:4859 , piece_id=4TIPWOWS7RI3JMXEIOMSUHK75C47BPQHOREVGXXDU4BHDJ6GO2JQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671570935 (211h41m0.102651066s) [noise=true , address=104.36.82.4:4811 , piece_id=IMUDMMGRKVVMD5NGLJOXX4CBAFP7WNZDIM2CL5AW43CENCK7JJTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671637472 (211h41m0.102717603s) [noise=true , address=104.36.82.23:4473 , piece_id=XJSZI5VY3SZBHIHMJHN5RIPKRQ4KF6JPL7YRPB6Q3RYQD5CHCXVQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671688780 (211h41m0.102768911s) [noise=true , address=104.36.82.35:4589 , piece_id=YCDME5SC67MHNTXSF6FP7ZP6AE5WIBL7HJU2X2MYF4PIBM6C3BDQ, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671718046 (211h41m0.102798167s) [noise=true , address=104.36.82.39:4738 , piece_id=HNKNIKTP6MBINHRYGIZCE37WSJZFVJJS3VUEFG5BM2AMAHCQR44Q, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:104.36.82.39:4738): protocol: write tcp 188.166.225.170:54906->104.36.82.39:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671778922 (211h41m0.102859053s) [noise=true , address=104.36.82.60:4340 , piece_id=CHEE76N5BQ7CR2I4KNK5EO5L22WFKHRZHAWHHSQX6DKONRKWECTA, seg_pos={0 0}, upload=220155]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:07:05.671819250 (211h41m0.102899381s) [seg_pos={0 0}, upload=220155]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 13:07:05.898779871 (211h41m0.329860042s) [upload=220155]: Done uploading segments... Aug 17 13:07:05.898803757 (211h41m0.329883888s) [upload=220155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.674629454s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)