Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXX.X.X.XXXXX..X.X...............X...X.......XX.....X.....] duration: 6.056123146s FAIL (2025-09-22 03:15:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.056158435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.062362993s FAIL (2025-09-22 03:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.062401336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXX...X..........................................XX........] duration: 6.071558622s FAIL (2025-09-22 03:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.071593069s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XXX.X.X.XXXXX..X.X...............X...X.......XX.....X....] duration: 5.953264177s FAIL (2025-09-22 03:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.953290947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXX...X..........................................XX.......] duration: 6.147509074s FAIL (2025-09-22 03:14:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.147537339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.961615869s FAIL (2025-09-22 03:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.961652008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXX...X..........................................XX......] duration: 5.993984175s FAIL (2025-09-22 03:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.994020155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XXX.X.X.XXXXX..X.X...............X...X.......XX.....X...] duration: 6.020909039s FAIL (2025-09-22 03:13:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.02094119s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...XXX...X..........................................XX.....] duration: 6.183503377s FAIL (2025-09-22 03:13:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.183543254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.975095684s FAIL (2025-09-22 03:13:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.975127443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....XXX.X.X.XXXXX..X.X...............X...X.......XX.....X..] duration: 6.051796296s FAIL (2025-09-22 03:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.051819831s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...XXX...X..........................................XX....] duration: 5.973931607s FAIL (2025-09-22 03:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.973963126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.871165451s FAIL (2025-09-22 03:12:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.871197622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.....XXX.X.X.XXXXX..X.X...............X...X.......XX.....X.] duration: 5.905948261s FAIL (2025-09-22 03:12:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905976154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X...XXX...X..........................................XX...] duration: 6.030868485s FAIL (2025-09-22 03:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.030900106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.936343142s FAIL (2025-09-22 03:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.936401743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.....XXX.X.X.XXXXX..X.X...............X...X.......XX.....X] duration: 30.001285771s FAIL (2025-09-22 03:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001321889s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.8998336s FAIL (2025-09-22 03:11:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.899861144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X...XXX...X..........................................XX..] duration: 5.89749055s FAIL (2025-09-22 03:11:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897524684s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X.X...XXX...X..........................................XX.] duration: 6.072939337s FAIL (2025-09-22 03:10:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.072961238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X.X...XXX...X..........................................XX] duration: 30.250939689s FAIL (2025-09-22 03:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:10:08.594159207 (373h41m44.633732337s) [upload=328468]: Starting upload Sep 22 03:10:08.594288313 (373h41m44.633861713s) [upload=328468]: Done starting upload Sep 22 03:10:08.594518761 (373h41m44.634091881s) [upload=328468]: Uploading segments... Sep 22 03:10:08.594565360 (373h41m44.63413849s) [upload=328468]: Got next segment. Inline: false Sep 22 03:10:08.594595398 (373h41m44.634168528s) [upload=328468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:10:08.594601048 (373h41m44.634174178s) [upload=328468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:10:08.856455116 (373h41m44.896028246s) [seg_pos={0 0}, upload=328468]: Begin upload segment... Sep 22 03:10:08.856516302 (373h41m44.896089422s) [seg_pos={0 0}, upload=328468]: Done begin upload segment. Sep 22 03:10:08.856553563 (373h41m44.896126683s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856559615 (373h41m44.896132735s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.856970497 (373h41m44.896543627s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857036673 (373h41m44.896609803s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857367854 (373h41m44.896940974s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.857721037 (373h41m44.897294157s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.858151806 (373h41m44.897724927s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.860387054 (373h41m44.899960174s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.861918790 (373h41m44.90149192s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862419964 (373h41m44.901993094s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862708574 (373h41m44.902281704s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.862991924 (373h41m44.902565054s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863312654 (373h41m44.902885774s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863579032 (373h41m44.903152152s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.863868974 (373h41m44.903442104s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:08.864308691 (373h41m44.903881821s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Uploading piece... Sep 22 03:10:10.563628285 (373h41m46.603201415s) [upload=328468]: Next returned nil segment Sep 22 03:10:10.563650588 (373h41m46.603223718s) [upload=328468]: Waiting for error group managing segments... Sep 22 03:10:10.959417891 (373h41m46.998991011s) [noise=true , address=77.90.31.19:4842 , piece_id=INRXVI4E2N77SJ2MG4RN3GG5SFES56NZF73FAEYHPUYFGZDSCLLA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.966844222 (373h41m47.006417352s) [noise=true , address=77.90.31.14:4663 , piece_id=E7UW3UBEFMUOODWWJLCVHDFGKD2YKIWXG2GV6MMUFMJ3P6TYS34A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.978792314 (373h41m47.018365444s) [noise=true , address=45.91.116.6:4974 , piece_id=3MD6UJTAKT3LZZG476TXPE6JHEGMSXSB5V35SCQFOHJVRAWLYHXA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.987680658 (373h41m47.027253789s) [noise=true , address=77.90.31.15:4699 , piece_id=Q4BIINGKAYYST7J445NK5DJZQE7MVKYQUFBL5XKD37EYM3OWWLPQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.997077572 (373h41m47.036650702s) [noise=true , address=45.91.116.4:4938 , piece_id=WWEO522WGMNHVE5JBIBDHATDFAD4NTOETJPNN64OA7YB6TAHBORA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:10.998045315 (373h41m47.037618445s) [noise=true , address=77.90.31.21:4879 , piece_id=IFQEVFV26YW6WOLEWNOV23JU3AXISRF5ZIK3YJKQOJ3QSVDNDUXQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002073846 (373h41m47.041646976s) [noise=true , address=77.90.31.12:4592 , piece_id=XBQSTHCFMHURWA7FNLTM3M7VCIH4YVLN7J2BX76HZFYBE5CV6XCA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.002897674 (373h41m47.042470804s) [noise=true , address=77.90.31.11:4553 , piece_id=FSHX7A6FRRSDYBNOLX34FPOQMXMTYTG3YHC67MY2GHOG45T6O6WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.005167517 (373h41m47.044740637s) [noise=true , address=77.90.31.17:4772 , piece_id=CVKUCFNK7PLSBCMPFKXL5JX2HCJRV5RJY2L7UU7GIK3H7T6D7LGA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.015113095 (373h41m47.054686215s) [noise=true , address=45.91.116.5:4297 , piece_id=3TEPLGQFFWFSHPW5OXT6PMOUECXUOTTSUCU5MPZMDXAGA4LME5PQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.044202085 (373h41m47.083775215s) [noise=true , address=77.90.31.16:4734 , piece_id=LZFRVKMAXB3BAXF2ZM4TJPSXNNAGBSMXAJAEPKDET36XE5OPQ2WA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:11.081472174 (373h41m47.121045304s) [noise=true , address=77.90.31.18:4807 , piece_id=3SXXD5NB47SW4VPBOPFTI2LNNFFJ5IT4NY3B3JBYXIAT4NQZM6UA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: <nil> Sep 22 03:10:25.795124945 (373h42m1.834698075s) [noise=true , address=145.79.227.16:4912 , piece_id=AOCME2ZMSMRY5A5TGK2ZPLQ3GWNW54GE7OQG6LWWVH5UGU6SOIFQ, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 03:10:37.878832559 (373h42m13.918405699s) [noise=true , address=145.79.227.5:4222 , piece_id=XKFQQY34I7Z3MFT62CIPDDXNBRNYK2HCEN6T6E2JPQRF44L6HY6A, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op Sep 22 03:10:37.879080240 (373h42m13.91865337s) [noise=true , address=145.79.227.17:4971 , piece_id=TGNNQY4RSWKMP27MZIRVUH75RPAXZCAWAVRMPEFVDRPSJPSYNF2Q, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op Sep 22 03:10:37.879325998 (373h42m13.918899118s) [noise=true , address=145.79.227.5:4223 , piece_id=YGRRVON7YM23QPDP72CHFMNCQBX2LCVIOVMQE2QZV23HJOD2IVEA, seg_pos={0 0}, upload=328468]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:10:37.879380893 (373h42m13.918954013s) [seg_pos={0 0}, upload=328468]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:10:38.127509645 (373h42m14.167082785s) [upload=328468]: Done uploading segments... Sep 22 03:10:38.127548429 (373h42m14.167121559s) [upload=328468]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.250970247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.116736955s FAIL (2025-09-22 03:10:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.116770999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.....XXX.X.X.XXXXX..X.X...............X...X.......XX.....] duration: 6.148846616s FAIL (2025-09-22 03:10:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.1491474s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.....XXX.X.X.XXXXX..X.X...............X...X.......XX....] duration: 6.418641444s FAIL (2025-09-22 03:09:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.418674709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.161693531s FAIL (2025-09-22 03:09:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.161731904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X.X...XXX...X..........................................X] duration: 30.243025995s FAIL (2025-09-22 03:09:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 03:08:53.664396589 (373h40m29.703969709s) [upload=328449]: Starting upload Sep 22 03:08:53.664476171 (373h40m29.704049291s) [upload=328449]: Done starting upload Sep 22 03:08:53.664644832 (373h40m29.704217952s) [upload=328449]: Uploading segments... Sep 22 03:08:53.664888406 (373h40m29.704461536s) [upload=328449]: Got next segment. Inline: false Sep 22 03:08:53.664973819 (373h40m29.704546949s) [upload=328449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:08:53.664980281 (373h40m29.704553411s) [upload=328449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:08:53.951587115 (373h40m29.991160235s) [seg_pos={0 0}, upload=328449]: Begin upload segment... Sep 22 03:08:53.951729175 (373h40m29.991302305s) [noise=true , address=45.91.116.8:4047 , piece_id=6RWYK2WBLS2PE3I7I42KDCRAPNEEDLD6UCTEEHPPDLF2OD4HRBFA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.952532165 (373h40m29.992105295s) [noise=true , address=77.90.31.15:4698 , piece_id=WTKVRRZ3JBM5IVW5RLWNV77UATZIU7G3YHTLBJ4TYRWJ5FOYBJIA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.952967483 (373h40m29.992540603s) [noise=true , address=45.91.116.11:4119 , piece_id=ARWFCJQNS3ZJYK5D47HRFEHHY54ZLWYJEJEU4Z7ZBEN4I6FXCASA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.953380590 (373h40m29.99295372s) [seg_pos={0 0}, upload=328449]: Done begin upload segment. Sep 22 03:08:53.953401950 (373h40m29.99297507s) [noise=true , address=145.79.227.6:4257 , piece_id=FN32GFNKJYE5IIN25YAD7DN7T2Y3ORWUH4GTJ63BSZMO245YTIYA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.954546199 (373h40m29.994119329s) [noise=true , address=145.79.227.3:4149 , piece_id=PYY6GFGHXQUR5Y6OUS4323J4762IQBMG4URGOQSM5DAL5V4PDJXA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.954913299 (373h40m29.994486419s) [noise=true , address=77.90.31.19:4843 , piece_id=SW4MO2Q56RH3TE3AKO7AZV7GX5ZWU6ZH4BXTZLJ7KKN65SVM76DQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.955199483 (373h40m29.994772603s) [noise=true , address=77.90.31.13:4626 , piece_id=H5MW23UA3AO6MPMYCLUAJT7W75IAXU53TU5BT2VBORS2SIQZF2SQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.955512059 (373h40m29.995085189s) [noise=true , address=45.91.116.10:4082 , piece_id=23GPXKCWWGLJPVEOPYEPSZ7434GQVHARUOB62WZYZI4NRRO6REOA, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.955821809 (373h40m29.995394929s) [noise=true , address=77.90.31.16:4733 , piece_id=KO6TJW6K3OXQKSQ57ZGG3UCRTE2SWX5D2OVOZM5NV4MSMFRO4ELQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.956192014 (373h40m29.995765144s) [noise=true , address=145.79.227.2:4114 , piece_id=ITRJSKQMNIY3ZH7NACWXWKBFN43J2OFJ2KYWKEBS4KNWWXQRA36Q, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.956470404 (373h40m29.996043534s) [noise=true , address=45.91.116.6:4973 , piece_id=K7UYNTRDTXOODDHTEX34PPFGTLK7XOL366WC46ENVEPZIW6HWS6A, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.958239412 (373h40m29.997812542s) [noise=true , address=45.91.116.6:4975 , piece_id=BNUVBARWVU6Z63ZVN5VRYCKPEB4V6AJ4KE3PBTTFNHMHM4RTOURQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.958531088 (373h40m29.998104208s) [noise=true , address=145.79.227.14:4837 , piece_id=J6VKYBJD5JKNFHBBCDYNAR3IGMY4RDED2BA2WKJDJN4EEVSLEEEQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.958787345 (373h40m29.998360475s) [noise=true , address=145.79.227.17:4970 , piece_id=HJJON3JPWQ2RF72XYEUHRQIGKX3YL2PA3B36PICON56DE6YZUZ4Q, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.959116913 (373h40m29.998690043s) [noise=true , address=45.91.116.14:4189 , piece_id=JRMVNSJX46L3EUDKTMSDRGNJCUSNRVV6A6CLZFGNWURB3RABUBGQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:53.959545559 (373h40m29.999118689s) [noise=true , address=77.90.31.12:4592 , piece_id=B45J6GZQUTJ3NENCQRC7RYHL7LBPYZR4M7OCYQXF22WFMLJMNDMQ, seg_pos={0 0}, upload=328449]: Uploading piece... Sep 22 03:08:55.740685374 (373h40m31.780258494s) [upload=328449]: Next returned nil segment Sep 22 03:08:55.740708508 (373h40m31.780281628s) [upload=328449]: Waiting for error group managing segments... Sep 22 03:08:56.024904529 (373h40m32.064477649s) [noise=true , address=45.91.116.11:4119 , piece_id=ARWFCJQNS3ZJYK5D47HRFEHHY54ZLWYJEJEU4Z7ZBEN4I6FXCASA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.027000801 (373h40m32.066573921s) [noise=true , address=77.90.31.15:4698 , piece_id=WTKVRRZ3JBM5IVW5RLWNV77UATZIU7G3YHTLBJ4TYRWJ5FOYBJIA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.037402828 (373h40m32.076975958s) [noise=true , address=45.91.116.6:4975 , piece_id=BNUVBARWVU6Z63ZVN5VRYCKPEB4V6AJ4KE3PBTTFNHMHM4RTOURQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.044407065 (373h40m32.083980195s) [noise=true , address=45.91.116.10:4082 , piece_id=23GPXKCWWGLJPVEOPYEPSZ7434GQVHARUOB62WZYZI4NRRO6REOA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.046784914 (373h40m32.086358044s) [noise=true , address=45.91.116.8:4047 , piece_id=6RWYK2WBLS2PE3I7I42KDCRAPNEEDLD6UCTEEHPPDLF2OD4HRBFA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.056586278 (373h40m32.096159408s) [noise=true , address=45.91.116.14:4189 , piece_id=JRMVNSJX46L3EUDKTMSDRGNJCUSNRVV6A6CLZFGNWURB3RABUBGQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.069781994 (373h40m32.109355125s) [noise=true , address=77.90.31.19:4843 , piece_id=SW4MO2Q56RH3TE3AKO7AZV7GX5ZWU6ZH4BXTZLJ7KKN65SVM76DQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.085008648 (373h40m32.124581768s) [noise=true , address=77.90.31.16:4733 , piece_id=KO6TJW6K3OXQKSQ57ZGG3UCRTE2SWX5D2OVOZM5NV4MSMFRO4ELQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.102699695 (373h40m32.142272825s) [noise=true , address=77.90.31.12:4592 , piece_id=B45J6GZQUTJ3NENCQRC7RYHL7LBPYZR4M7OCYQXF22WFMLJMNDMQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.117355313 (373h40m32.156928443s) [noise=true , address=77.90.31.13:4626 , piece_id=H5MW23UA3AO6MPMYCLUAJT7W75IAXU53TU5BT2VBORS2SIQZF2SQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:08:56.141442900 (373h40m32.18101603s) [noise=true , address=45.91.116.6:4973 , piece_id=K7UYNTRDTXOODDHTEX34PPFGTLK7XOL366WC46ENVEPZIW6HWS6A, seg_pos={0 0}, upload=328449]: Done uploading piece. err: <nil> Sep 22 03:09:22.970208391 (373h40m59.009781522s) [noise=true , address=145.79.227.6:4257 , piece_id=FN32GFNKJYE5IIN25YAD7DN7T2Y3ORWUH4GTJ63BSZMO245YTIYA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:09:22.970295286 (373h40m59.009868416s) [noise=true , address=145.79.227.2:4114 , piece_id=ITRJSKQMNIY3ZH7NACWXWKBFN43J2OFJ2KYWKEBS4KNWWXQRA36Q, seg_pos={0 0}, upload=328449]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:09:22.970351043 (373h40m59.009924173s) [noise=true , address=145.79.227.3:4149 , piece_id=PYY6GFGHXQUR5Y6OUS4323J4762IQBMG4URGOQSM5DAL5V4PDJXA, seg_pos={0 0}, upload=328449]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:09:22.970398643 (373h40m59.009971763s) [noise=true , address=145.79.227.17:4970 , piece_id=HJJON3JPWQ2RF72XYEUHRQIGKX3YL2PA3B36PICON56DE6YZUZ4Q, seg_pos={0 0}, upload=328449]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:09:22.970514294 (373h40m59.010087424s) [noise=true , address=145.79.227.14:4837 , piece_id=J6VKYBJD5JKNFHBBCDYNAR3IGMY4RDED2BA2WKJDJN4EEVSLEEEQ, seg_pos={0 0}, upload=328449]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:09:22.970555302 (373h40m59.010128432s) [seg_pos={0 0}, upload=328449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 03:09:23.211059224 (373h40m59.250632354s) [upload=328449]: Done uploading segments... Sep 22 03:09:23.211095473 (373h40m59.250668593s) [upload=328449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.243068136s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.....XXX.X.X.XXXXX..X.X...............X...X.......XX...] duration: 6.151853121s FAIL (2025-09-22 03:09:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.151891433s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.....XXX.X.X.XXXXX..X.X...............X...X.......XX..] duration: 5.90945528s FAIL (2025-09-22 03:08:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.909724003s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.814754623s FAIL (2025-09-22 03:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.814787936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X.X.X...XXX...X..........................................] duration: 5.952441213s FAIL (2025-09-22 03:08:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.952482863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X.X.X...XXX...X.........................................] duration: 5.975735091s FAIL (2025-09-22 03:07:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.97576119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.965800092s FAIL (2025-09-22 03:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.965834167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.....XXX.X.X.XXXXX..X.X...............X...X.......XX.] duration: 6.150711628s FAIL (2025-09-22 03:07:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.150742376s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X.....XXX.X.X.XXXXX..X.X...............X...X.......XX] duration: 30.004355088s FAIL (2025-09-22 03:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.004384363s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.891637505s FAIL (2025-09-22 03:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.891660319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX..X.X.X...XXX...X........................................] duration: 6.232521459s FAIL (2025-09-22 03:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.232563048s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........X.....XXX.X.X.XXXXX..X.X...............X...X.......X] duration: 30.00036878s FAIL (2025-09-22 03:06:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000392584s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.XX..X.X.X...XXX...X.......................................] duration: 5.992060239s FAIL (2025-09-22 03:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.992100526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.678993835s FAIL (2025-09-22 03:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.679026406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.109909842s FAIL (2025-09-22 03:05:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.109943927s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X.....XXX.X.X.XXXXX..X.X...............X...X.......] duration: 6.312246975s FAIL (2025-09-22 03:05:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.312279106s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX.XX..X.X.X...XXX...X......................................] duration: 6.10483315s FAIL (2025-09-22 03:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.1048748s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.........X.....XXX.X.X.XXXXX..X.X...............X...X......] duration: 6.14198541s FAIL (2025-09-22 03:04:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.142013031s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.........X.....XXX.X.X.XXXXX..X.X...............X...X.....] duration: 6.373360728s FAIL (2025-09-22 03:04:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.373390907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.786672633s FAIL (2025-09-22 03:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 5.786717158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.201279868s FAIL (2025-09-22 03:04:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.201317179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XX.XX..X.X.X...XXX...X.....................................] duration: 6.038672916s FAIL (2025-09-22 03:03:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.038713905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.021126405s FAIL (2025-09-22 03:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== g upload Sep 22 03:01:05.865553220 (373h32m41.90512635s) [upload=328333]: Done starting upload Sep 22 03:01:05.865704920 (373h32m41.90527806s) [upload=328333]: Uploading segments... Sep 22 03:01:05.865879602 (373h32m41.905452733s) [upload=328333]: Got next segment. Inline: false Sep 22 03:01:05.865956689 (373h32m41.905529819s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:01:05.865964033 (373h32m41.905537163s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:01:06.344240899 (373h32m42.383814029s) [seg_pos={0 0}, upload=328333]: Begin upload segment... Sep 22 03:01:06.344323377 (373h32m42.383896507s) [seg_pos={0 0}, upload=328333]: Done begin upload segment. Sep 22 03:01:06.344357432 (373h32m42.383930572s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344440439 (373h32m42.384013559s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.344898792 (373h32m42.384471922s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345280158 (373h32m42.384853288s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345674760 (373h32m42.38524789s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.345746146 (373h32m42.385319276s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346112974 (373h32m42.385686104s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346495192 (373h32m42.386068332s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.346852843 (373h32m42.386425973s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347221756 (373h32m42.386794886s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.347602090 (373h32m42.38717522s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348012181 (373h32m42.387585311s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348408345 (373h32m42.387981475s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.348781977 (373h32m42.388355107s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349096025 (373h32m42.388669165s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.349625593 (373h32m42.389198733s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351073210 (373h32m42.39064635s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:06.351572460 (373h32m42.3911456s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Uploading piece... Sep 22 03:01:07.836111554 (373h32m43.875684684s) [upload=328333]: Next returned nil segment Sep 22 03:01:07.836135158 (373h32m43.875708288s) [upload=328333]: Waiting for error group managing segments... Sep 22 03:01:08.234359500 (373h32m44.27393263s) [noise=true , address=145.79.227.20:4455 , piece_id=675LI3TTHGCRSPA25KAUMLYO4TBYZSKR674B2S6JT3WLRSOCR7MQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.237279180 (373h32m44.27685231s) [noise=true , address=145.79.227.46:4624 , piece_id=LZTDJHZGA72MWPU6GVIFYESSFSRUQEJ4CURNNDAAZLVWYAJMFE7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.238462203 (373h32m44.278035333s) [noise=true , address=145.79.227.35:4508 , piece_id=3U24EY7HAUOZVRIAXMCPEA4UN2WPMSQLBUN3DTEAWY5AZ2TFCHDA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239241416 (373h32m44.278814546s) [noise=true , address=145.79.227.28:4755 , piece_id=N4NFOX7CQNDFWF66LSV6UEXVRNSU4HTMAQWCSQSHA32GZHFU6VUQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.239482235 (373h32m44.279055355s) [noise=true , address=145.79.227.52:4465 , piece_id=OGGJWOQOBGNJSTMPJXMTVUPVUNBQD4P4E35RKX5AL66ECEPTAZIQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245212557 (373h32m44.284785687s) [noise=true , address=145.79.227.25:4625 , piece_id=VQESIR5IDELAKTMVZMDFD6JYIWNHUHC4AZRFIFDXM63O43SOTSIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245377651 (373h32m44.284950781s) [noise=true , address=145.79.227.31:4802 , piece_id=6GXUSRDGEMMJBBBIOI4ZFSUQTYE47IU7CJ6DBPI62SMQQJZZP6EA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.245580787 (373h32m44.285153918s) [noise=true , address=145.79.227.47:4284 , piece_id=7OC7VXEDXO2J3C6NXO2VKQ4JBLAGJDNHD3LBIIGWHPI6U77BFVHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.246303674 (373h32m44.285876804s) [noise=true , address=145.79.227.32:4862 , piece_id=BEGNIJMZOXT5DQ3LCIM2GMUSIKIUZGYPA37JBJXBA2DULE65HRIA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.247775527 (373h32m44.287348657s) [noise=true , address=145.79.227.23:4571 , piece_id=HRONCJ3AQKAGARJUQ7JNLHJ44ZEKZCFDYH6JJJNZFG4PJI6FF6YQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.248450091 (373h32m44.288023221s) [noise=true , address=145.79.227.33:4875 , piece_id=X6CKD5EVX4EJ7SQF7BMN4FCHEDJW2A224B4DF54MDXDDU3KZKZ7Q, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.251320647 (373h32m44.290893777s) [noise=true , address=145.79.227.22:4526 , piece_id=SI6FUNVBWIESHXNAHMRE3Y2LOTCIQTQ3DOHBVW2B3VO7MV6B5QBA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.256449654 (373h32m44.296022774s) [noise=true , address=145.79.227.41:4095 , piece_id=GTKJJEUEDBSZKTFYD2FVJUWQIVNQDUP4A63ZVGMWVN7NSIBWFTHQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296203442 (373h32m44.335776572s) [noise=true , address=145.79.227.49:4346 , piece_id=LV2SEQRRJ5NQKUUMFNNWT6A7HNBGI34CGXYJRXN5QYIZFH2SDZWA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: <nil> Sep 22 03:01:08.296224112 (373h32m44.335797232s) [seg_pos={0 0}, upload=328333]: Segment reached optimal threshold of 14 pieces. Sep 22 03:01:08.296361023 (373h32m44.335934143s) [noise=true , address=77.90.31.18:4837 , piece_id=TK4MDK7GHZOQTKHM5USTRZ7DZ5XEHNQAVQILYQEUR7URX5FSJ5DA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296489046 (373h32m44.336062176s) [noise=true , address=77.90.31.19:4854 , piece_id=NNOKBRCN7FYIXADWCX2O4BBA346UVQKMZPA2EREY6IM5TVZVLDMQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296576182 (373h32m44.336149302s) [noise=true , address=77.90.31.10:4526 , piece_id=3RADBR5WK7XWXQVZFXCYMGS47G44UHJ7EWBIMWHAO6H7CY5KLZBQ, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296673327 (373h32m44.336246458s) [noise=true , address=77.90.31.16:4764 , piece_id=AJ774X5D32ZWNY7QFA5XSJOU4OLDARBGFZ2NPA6UUZ5THMC464AA, seg_pos={0 0}, upload=328333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Sep 22 03:01:08.296686632 (373h32m44.336259762s) [seg_pos={0 0}, upload=328333]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 22 03:01:08.296756606 (373h32m44.336329736s) [upload=328333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 22 03:01:08.296761445 (373h32m44.336334565s) [upload=328333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 22 03:01:08.653449354 (373h32m44.693022484s) [upload=328333]: Done uploading segments... Sep 22 03:01:08.653475584 (373h32m44.693048714s) [upload=328333]: Upload finished. err: <nil> Duration: 6.021161051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.........X.....XXX.X.X.XXXXX..X.X...............X...X....] duration: 5.941291832s FAIL (2025-09-22 03:03:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.941314635s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XX.XX..X.X.X...XXX...X....................................] duration: 6.106727605s FAIL (2025-09-22 03:03:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.106758634s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)