Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X] duration: 30.232289548s FAIL (2025-07-27 00:43:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== loading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:53950->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:07.609111953 (61h7m17.331674993s) [seg_pos={0 0}, upload=58605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:43:07.609198008 (61h7m17.331761038s) [upload=58605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:43:07.609202506 (61h7m17.331765546s) [upload=58605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:43:07.896913209 (61h7m17.619476249s) [upload=58605]: Done uploading segments... Jul 27 00:43:07.896931484 (61h7m17.619494524s) [upload=58605]: Upload finished. err: <nil> Jul 27 00:43:28.060523287 (61h7m37.783086327s) [upload=58612]: Starting upload Jul 27 00:43:28.060583983 (61h7m37.783147013s) [upload=58612]: Done starting upload Jul 27 00:43:28.060670480 (61h7m37.78323352s) [upload=58612]: Uploading segments... Jul 27 00:43:28.060757105 (61h7m37.783320135s) [upload=58612]: Got next segment. Inline: false Jul 27 00:43:28.060782454 (61h7m37.783345484s) [upload=58612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:28.060787423 (61h7m37.783350463s) [upload=58612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:28.290711414 (61h7m38.013274534s) [seg_pos={0 0}, upload=58612]: Begin upload segment... Jul 27 00:43:28.290781437 (61h7m38.013344487s) [seg_pos={0 0}, upload=58612]: Done begin upload segment. Jul 27 00:43:28.290806114 (61h7m38.013369154s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.290862703 (61h7m38.013425763s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291198386 (61h7m38.013761426s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291275082 (61h7m38.013838132s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291495654 (61h7m38.014058704s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291856834 (61h7m38.014419864s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291880410 (61h7m38.01444347s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292322225 (61h7m38.014885265s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292451232 (61h7m38.015014282s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292763219 (61h7m38.015326259s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292937924 (61h7m38.015500964s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293180598 (61h7m38.015743638s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293528122 (61h7m38.016091172s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293599289 (61h7m38.016162329s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.294088004 (61h7m38.016651044s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.298318578 (61h7m38.020881608s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:30.041308206 (61h7m39.763871246s) [upload=58612]: Next returned nil segment Jul 27 00:43:30.041324637 (61h7m39.763887677s) [upload=58612]: Waiting for error group managing segments... Jul 27 00:43:30.349307817 (61h7m40.071870857s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.375421520 (61h7m40.09798456s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.381080437 (61h7m40.103643477s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.411307720 (61h7m40.13387076s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.454531159 (61h7m40.177094199s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.499012215 (61h7m40.221575255s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.522553385 (61h7m40.245116425s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.553051998 (61h7m40.275615038s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:31.247387600 (61h7m40.96995063s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.097194573 (61h7m41.819757653s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.214252718 (61h7m41.936815758s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.025053300 (61h8m6.74761634s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.418787467 (61h8m7.141350507s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:35220->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418925340 (61h8m7.14148838s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:55892->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418975867 (61h8m7.141538907s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:52952->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418980546 (61h8m7.141543586s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:43978->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.419029480 (61h8m7.14159256s) [seg_pos={0 0}, upload=58612]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:43:57.648786891 (61h8m7.371349931s) [upload=58612]: Done uploading segments... Jul 27 00:43:57.648814473 (61h8m7.371377514s) [upload=58612]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232306892s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X......X.XX...XXXX.....X..........X..X..X...XX..........] duration: 7.529788539s FAIL (2025-07-27 00:43:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.529813285s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.] duration: 7.020350848s FAIL (2025-07-27 00:43:49 UTC) orbiter warning: uplink: context deadline exceeded Duration: 7.020436762s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...X......X.XX...XXXX.....X..........X..X..X...XX.........] duration: 5.83313763s FAIL (2025-07-27 00:43:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.833160594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX] duration: 30.002426766s FAIL (2025-07-27 00:43:15 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002452816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.] duration: 5.839658075s FAIL (2025-07-27 00:43:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== true , address=104.36.82.2:4732 , piece_id=X4V7K4DDWYN5ZOYJE4PG6MRZJKRYIDUHCHZDNVOEDK62AXCUE2SQ, seg_pos={0 0}, upload=58575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:49186->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:08.618647277 (61h5m18.341210317s) [seg_pos={0 0}, upload=58575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:41:08.618706270 (61h5m18.34126931s) [upload=58575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:41:08.618709977 (61h5m18.341273017s) [upload=58575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:41:09.066971850 (61h5m18.78953489s) [upload=58575]: Done uploading segments... Jul 27 00:41:09.066994283 (61h5m18.789557323s) [upload=58575]: Upload finished. err: <nil> Jul 27 00:41:59.867159718 (61h6m9.589722768s) [upload=58588]: Starting upload Jul 27 00:41:59.867245562 (61h6m9.589808602s) [upload=58588]: Done starting upload Jul 27 00:41:59.867321347 (61h6m9.589884387s) [upload=58588]: Uploading segments... Jul 27 00:41:59.867524265 (61h6m9.590087295s) [upload=58588]: Got next segment. Inline: false Jul 27 00:41:59.867586214 (61h6m9.590149264s) [upload=58588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:59.867591905 (61h6m9.590154945s) [upload=58588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:42:00.102737386 (61h6m9.825300416s) [seg_pos={0 0}, upload=58588]: Begin upload segment... Jul 27 00:42:00.102779776 (61h6m9.825342816s) [seg_pos={0 0}, upload=58588]: Done begin upload segment. Jul 27 00:42:00.102801278 (61h6m9.825364308s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.102846094 (61h6m9.825409124s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103240107 (61h6m9.825803147s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103336531 (61h6m9.825899571s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103580899 (61h6m9.826143939s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103650994 (61h6m9.826214034s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103936319 (61h6m9.826499359s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104330203 (61h6m9.826893243s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104636319 (61h6m9.827199349s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104885456 (61h6m9.827448496s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105121977 (61h6m9.827685017s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105330807 (61h6m9.827893847s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105423625 (61h6m9.827986665s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105691517 (61h6m9.828254557s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105926467 (61h6m9.828489507s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105996781 (61h6m9.828559821s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:01.832554780 (61h6m11.55511782s) [upload=58588]: Next returned nil segment Jul 27 00:42:01.832578957 (61h6m11.555142007s) [upload=58588]: Waiting for error group managing segments... Jul 27 00:42:02.113654143 (61h6m11.836217183s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.113875226 (61h6m11.836438266s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.122043925 (61h6m11.844606966s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.144804754 (61h6m11.867367804s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.194302797 (61h6m11.916865837s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.212140827 (61h6m11.934703867s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.215132661 (61h6m11.937695701s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.226526110 (61h6m11.94908915s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.239060171 (61h6m11.961623211s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.287304216 (61h6m12.009867256s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.290839160 (61h6m12.01340221s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.298282962 (61h6m12.020846002s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:29.221210346 (61h6m38.943773386s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221257307 (61h6m38.943820337s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221291121 (61h6m38.943854161s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221325297 (61h6m38.943888337s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:50082->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221339103 (61h6m38.943902133s) [seg_pos={0 0}, upload=58588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:42:29.461689803 (61h6m39.184252843s) [upload=58588]: Done uploading segments... Jul 27 00:42:29.461705864 (61h6m39.184268904s) [upload=58588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.839683474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233775466s FAIL (2025-07-27 00:43:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Done begin upload segment. Jul 27 00:42:04.785142655 (61h6m14.507705695s) [noise=true , address=104.36.82.10:4100 , piece_id=O46ZDV4F643AVLSDFAJIXVUTXC3JSCJU4FVZR34HYVOTPK42T26Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785202089 (61h6m14.507765129s) [noise=true , address=104.36.82.56:4258 , piece_id=76V3RW55QWH3Y3CH3IC5OCXBE6PI6W2CSTQ6C7GLYIT72ZP2HMUA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785252966 (61h6m14.507816006s) [noise=true , address=79.127.205.246:4262 , piece_id=LSFXKXWXK4Z5VORQS7IFOLSSNGXLJHPZZPP72ZH4VJML2MEI6VZQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785552720 (61h6m14.50811575s) [noise=true , address=104.36.82.46:4511 , piece_id=R3AVBVU3PPOOTT2IGTH3IDRZHCHTSZXHSZHRL4I5X3AYWXMWGSXQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785648724 (61h6m14.508211754s) [noise=true , address=79.127.219.38:4001 , piece_id=XAPBQVMSKM5WY734J7QKJHIMLLGEHZEDU63FGC3PDIVZJVFC53EQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785840571 (61h6m14.508403611s) [noise=true , address=104.36.82.54:4176 , piece_id=VUMPR5JXGY3QJ4UEAIA3WXFWOYJIGIBBAIHT2WQM6EVJJZZZF7XQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785932957 (61h6m14.508495997s) [noise=true , address=79.127.219.38:4015 , piece_id=DCGZ2CELQA2ZJLI32IRCYECTTCGAOG5XQYZSQAFLMDQQ76ORSPWQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786180421 (61h6m14.508743461s) [noise=true , address=64.49.13.139:4478 , piece_id=74U7AHILBJTTWXCSEFNQJ34SPTMPDXGDBM364ZTJ24XRQ7YNI3IA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786436851 (61h6m14.508999891s) [noise=true , address=77.90.31.18:4808 , piece_id=FHMJB43JS663HBVDMKBIIDOQW35WM76BU6HG7AAZMA4WAG3FQ5FQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786588912 (61h6m14.509151963s) [noise=true , address=79.127.219.38:4009 , piece_id=4CMD3MFUPNXO7DS4JSCWS774WQ5YB753IAJZEVQEWAMSWQ2PWF3Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786676690 (61h6m14.50923973s) [noise=true , address=104.36.82.52:4093 , piece_id=5EASC4U5X2LOW2LMLQAGZBNKXA7PW352AXE6DPF7NNSVC5ZXHAGA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786915047 (61h6m14.509478087s) [noise=true , address=104.36.82.53:4134 , piece_id=Q2ORRC2BJEM5LBGTTRIU4HXL73I7XVDVYUGDOJULWJUSDCTJK2AQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787046859 (61h6m14.509609889s) [noise=true , address=104.36.82.55:4224 , piece_id=JLNXUOBATLASOFJY2ZRDHBM4NSXTC4QRTUYJQZNWKP57NYXXRCMA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787106002 (61h6m14.509669042s) [noise=true , address=104.36.82.47:4521 , piece_id=JXILHMKJ4O6KRHLSEJQUYQ5Y2GCITGOO6NV7NJTKJJEQVFY3644A, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787310122 (61h6m14.509873162s) [noise=true , address=104.36.82.4:4819 , piece_id=ID5J72EHY47UDSEALG7GAON3Z24IBU53QEOZIFN6CNKKP4HR3BRA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787518351 (61h6m14.510081391s) [noise=true , address=79.127.205.245:4240 , piece_id=X2YYCYYNWHIFEYQRAQ5OH3QRSJY62I4F35IQD5VMLZV4GZQAKRBQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787716440 (61h6m14.51027948s) [noise=true , address=156.146.43.225:4448 , piece_id=T5PUJDQF6XXDGTYDYFT3GK5BBGJEH6737BRBOHDHNKDLSLZP7KBQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787962471 (61h6m14.510525511s) [noise=true , address=79.127.205.235:4686 , piece_id=6T7TIYC56I6XKHFQAK6FXGS3NVBHSPK5VIPDXQ22YB234VGDN33Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:05.821517206 (61h6m15.544080236s) [noise=true , address=156.146.43.225:4448 , piece_id=T5PUJDQF6XXDGTYDYFT3GK5BBGJEH6737BRBOHDHNKDLSLZP7KBQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:42:05.899180068 (61h6m15.621743108s) [upload=58589]: Next returned nil segment Jul 27 00:42:05.899197260 (61h6m15.6217603s) [upload=58589]: Waiting for error group managing segments... Jul 27 00:42:06.186741615 (61h6m15.909304655s) [noise=true , address=79.127.205.235:4686 , piece_id=6T7TIYC56I6XKHFQAK6FXGS3NVBHSPK5VIPDXQ22YB234VGDN33Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.200602645 (61h6m15.923165685s) [noise=true , address=79.127.205.246:4262 , piece_id=LSFXKXWXK4Z5VORQS7IFOLSSNGXLJHPZZPP72ZH4VJML2MEI6VZQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.230780924 (61h6m15.953343984s) [noise=true , address=79.127.205.245:4240 , piece_id=X2YYCYYNWHIFEYQRAQ5OH3QRSJY62I4F35IQD5VMLZV4GZQAKRBQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.697146484 (61h6m16.419709524s) [noise=true , address=77.90.31.18:4808 , piece_id=FHMJB43JS663HBVDMKBIIDOQW35WM76BU6HG7AAZMA4WAG3FQ5FQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.753883168 (61h6m16.476446208s) [noise=true , address=79.127.219.38:4015 , piece_id=DCGZ2CELQA2ZJLI32IRCYECTTCGAOG5XQYZSQAFLMDQQ76ORSPWQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.961489417 (61h6m16.684052457s) [noise=true , address=79.127.219.38:4001 , piece_id=XAPBQVMSKM5WY734J7QKJHIMLLGEHZEDU63FGC3PDIVZJVFC53EQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:07.002187174 (61h6m16.724750224s) [noise=true , address=79.127.219.38:4009 , piece_id=4CMD3MFUPNXO7DS4JSCWS774WQ5YB753IAJZEVQEWAMSWQ2PWF3Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:16.006986552 (61h6m25.729549622s) [noise=true , address=64.49.13.139:4478 , piece_id=74U7AHILBJTTWXCSEFNQJ34SPTMPDXGDBM364ZTJ24XRQ7YNI3IA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:00.424439591 (61h7m10.147002622s) [noise=true , address=104.36.82.46:4511 , piece_id=R3AVBVU3PPOOTT2IGTH3IDRZHCHTSZXHSZHRL4I5X3AYWXMWGSXQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:00.688562063 (61h7m10.411125103s) [noise=true , address=104.36.82.47:4521 , piece_id=JXILHMKJ4O6KRHLSEJQUYQ5Y2GCITGOO6NV7NJTKJJEQVFY3644A, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:03.618547383 (61h7m13.341110423s) [noise=true , address=104.36.82.52:4093 , piece_id=5EASC4U5X2LOW2LMLQAGZBNKXA7PW352AXE6DPF7NNSVC5ZXHAGA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618612206 (61h7m13.341175236s) [noise=true , address=104.36.82.4:4819 , piece_id=ID5J72EHY47UDSEALG7GAON3Z24IBU53QEOZIFN6CNKKP4HR3BRA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12mh9u38YmD7NcPA5uB1KLaw8dToTUKywJsi1MhhSc17LXo15Gb, address:104.36.82.4:4819): protocol: write tcp 188.166.225.170:39594->104.36.82.4:4819: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618641412 (61h7m13.341204452s) [noise=true , address=104.36.82.56:4258 , piece_id=76V3RW55QWH3Y3CH3IC5OCXBE6PI6W2CSTQ6C7GLYIT72ZP2HMUA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:45148->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618671069 (61h7m13.341234109s) [noise=true , address=104.36.82.54:4176 , piece_id=VUMPR5JXGY3QJ4UEAIA3WXFWOYJIGIBBAIHT2WQM6EVJJZZZF7XQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618710875 (61h7m13.341273915s) [noise=true , address=104.36.82.53:4134 , piece_id=Q2ORRC2BJEM5LBGTTRIU4HXL73I7XVDVYUGDOJULWJUSDCTJK2AQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618720934 (61h7m13.341283974s) [noise=true , address=104.36.82.55:4224 , piece_id=JLNXUOBATLASOFJY2ZRDHBM4NSXTC4QRTUYJQZNWKP57NYXXRCMA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618730884 (61h7m13.341293924s) [noise=true , address=104.36.82.10:4100 , piece_id=O46ZDV4F643AVLSDFAJIXVUTXC3JSCJU4FVZR34HYVOTPK42T26Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618744980 (61h7m13.34130802s) [seg_pos={0 0}, upload=58589]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:43:03.850086514 (61h7m13.572649554s) [upload=58589]: Done uploading segments... Jul 27 00:43:03.850111922 (61h7m13.572674962s) [upload=58589]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.233791669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.852844591s FAIL (2025-07-27 00:43:01 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.852877474s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.X.XX...X...............XX......X...XXX..XXX..X..X.X.......] duration: 5.486130424s FAIL (2025-07-27 00:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.486180611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XX.X.XX...X...............XX......X...XXX..XXX..X..X.X......] duration: 16.542392525s FAIL (2025-07-27 00:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.542416792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXX] duration: 30.26796102s FAIL (2025-07-27 00:42:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.267984837s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X...X......X.XX...XXXX.....X..........X..X..X...XX........] duration: 5.938656788s FAIL (2025-07-27 00:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938679451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X] duration: 30.242376621s FAIL (2025-07-27 00:42:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== true , address=104.36.82.2:4732 , piece_id=X4V7K4DDWYN5ZOYJE4PG6MRZJKRYIDUHCHZDNVOEDK62AXCUE2SQ, seg_pos={0 0}, upload=58575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:49186->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:08.618647277 (61h5m18.341210317s) [seg_pos={0 0}, upload=58575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:41:08.618706270 (61h5m18.34126931s) [upload=58575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:41:08.618709977 (61h5m18.341273017s) [upload=58575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:41:09.066971850 (61h5m18.78953489s) [upload=58575]: Done uploading segments... Jul 27 00:41:09.066994283 (61h5m18.789557323s) [upload=58575]: Upload finished. err: <nil> Jul 27 00:41:59.867159718 (61h6m9.589722768s) [upload=58588]: Starting upload Jul 27 00:41:59.867245562 (61h6m9.589808602s) [upload=58588]: Done starting upload Jul 27 00:41:59.867321347 (61h6m9.589884387s) [upload=58588]: Uploading segments... Jul 27 00:41:59.867524265 (61h6m9.590087295s) [upload=58588]: Got next segment. Inline: false Jul 27 00:41:59.867586214 (61h6m9.590149264s) [upload=58588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:59.867591905 (61h6m9.590154945s) [upload=58588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:42:00.102737386 (61h6m9.825300416s) [seg_pos={0 0}, upload=58588]: Begin upload segment... Jul 27 00:42:00.102779776 (61h6m9.825342816s) [seg_pos={0 0}, upload=58588]: Done begin upload segment. Jul 27 00:42:00.102801278 (61h6m9.825364308s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.102846094 (61h6m9.825409124s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103240107 (61h6m9.825803147s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103336531 (61h6m9.825899571s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103580899 (61h6m9.826143939s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103650994 (61h6m9.826214034s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103936319 (61h6m9.826499359s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104330203 (61h6m9.826893243s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104636319 (61h6m9.827199349s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104885456 (61h6m9.827448496s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105121977 (61h6m9.827685017s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105330807 (61h6m9.827893847s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105423625 (61h6m9.827986665s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105691517 (61h6m9.828254557s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105926467 (61h6m9.828489507s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105996781 (61h6m9.828559821s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:01.832554780 (61h6m11.55511782s) [upload=58588]: Next returned nil segment Jul 27 00:42:01.832578957 (61h6m11.555142007s) [upload=58588]: Waiting for error group managing segments... Jul 27 00:42:02.113654143 (61h6m11.836217183s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.113875226 (61h6m11.836438266s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.122043925 (61h6m11.844606966s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.144804754 (61h6m11.867367804s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.194302797 (61h6m11.916865837s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.212140827 (61h6m11.934703867s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.215132661 (61h6m11.937695701s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.226526110 (61h6m11.94908915s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.239060171 (61h6m11.961623211s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.287304216 (61h6m12.009867256s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.290839160 (61h6m12.01340221s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.298282962 (61h6m12.020846002s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:29.221210346 (61h6m38.943773386s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221257307 (61h6m38.943820337s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221291121 (61h6m38.943854161s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221325297 (61h6m38.943888337s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:50082->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221339103 (61h6m38.943902133s) [seg_pos={0 0}, upload=58588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:42:29.461689803 (61h6m39.184252843s) [upload=58588]: Done uploading segments... Jul 27 00:42:29.461705864 (61h6m39.184268904s) [upload=58588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242393824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.859516064s FAIL (2025-07-27 00:42:18 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.859549629s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230154452s FAIL (2025-07-27 00:42:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 9.13.26:4953 , piece_id=5Y4IGZPA6IVL3KBNBI4AMVRZ4LBMVNAMB6ZESZ5QUGUHUEAHSS4A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281505415 (61h5m14.004068455s) [noise=true , address=77.90.31.17:4798 , piece_id=NBI6QAZEMYHWCD5Q7PIHK3VKDFMCKZEG5CCCKQ2ZQEZ6ZHZIJ3RA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281590748 (61h5m14.004153788s) [noise=true , address=156.146.43.225:4449 , piece_id=CEP3OYAT54MORXPOFR7J7ZSZ2VVS4HX6EJXF7PAJ34ASULHAERUA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281779188 (61h5m14.004342229s) [noise=true , address=79.127.205.233:4652 , piece_id=DQAKEWUAPGO7LIYGWB4GPNGRAXAQDAQEQFYIORV5VFPL6RNCDHTQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282059434 (61h5m14.004622475s) [noise=true , address=104.36.82.47:4533 , piece_id=AS5YO7FDM5LCMAK2JBJF6F5YJUPMS6E2P3U5TE5ATTETYJH3KMWA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282342457 (61h5m14.004905497s) [noise=true , address=79.127.201.215:4358 , piece_id=B5A2TJE7DNKIB3OOILCUGJ5R6MT4T34MYUDELYUVQQTIHGG4GNQQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282631549 (61h5m14.005194589s) [noise=true , address=77.90.31.18:4808 , piece_id=RZLEG2RVUXSMQTVXPN4B6NGDD5ZB7DWWTDIIQ4NKJJ2BOUQZ5S2A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282854095 (61h5m14.005417135s) [noise=true , address=77.90.31.21:4906 , piece_id=EGX353TKU2ZDXQ4NBOINI3OP7FDJU4OFNHWV3NKFXSIKPJV5E4QA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283059278 (61h5m14.005622318s) [noise=true , address=104.36.82.46:4511 , piece_id=VNUMKF4OMIJRIH4UP2URJPZ4I2I7YZ62RW6I4UYVO5Y54GRFCHVA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283279299 (61h5m14.005842329s) [noise=true , address=104.36.82.53:4121 , piece_id=NJNJ4DJEZ54JW3SE74WOMPGSOTITSCF2CLWNB36MHWBSC63FMQGA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283496164 (61h5m14.006059194s) [noise=true , address=79.127.201.215:4356 , piece_id=J26JGG4UOVETDXUVYRJR742W7FYEPZS2G2PMF2OO5WWVFN75LPNQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283696136 (61h5m14.006259177s) [noise=true , address=77.90.31.16:4746 , piece_id=5XL5RQKYILX3SST5ZRW6BOVMDPHBWIWK6D25HKMXBH2XQREKH3GQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283952477 (61h5m14.006515507s) [noise=true , address=79.127.205.235:4676 , piece_id=DL6OOBP52DR5H2MLV5JC6GVL2RTSMBBAPJJN2QCSWRWEF5ET3IPA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284157159 (61h5m14.006720199s) [noise=true , address=79.127.205.245:4242 , piece_id=GW3T62S6N6UHKIAEICIUTYDZD7SEPTYIIFGF2MCZZORKGJK26IKA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284369995 (61h5m14.006933025s) [noise=true , address=104.36.82.54:4177 , piece_id=VUJ3AA7YIPUMKUFZBYVWGJQ5O3WKMQ5NZQGTHL4XSUSGZ7OVSVNQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284578745 (61h5m14.007141775s) [noise=true , address=104.36.82.12:4173 , piece_id=UUTD5HXRZRQ4EQRGAL7BFBDCFZCPJVJLNKL5LUPEOZ54XYBNSO3A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284795059 (61h5m14.007358099s) [noise=true , address=104.36.82.50:4048 , piece_id=OCDELTG5CJVE5ZLFP73LIZCHOFOGFOFOVR3TQTXO5TI2DGXNU2KQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:05.386679138 (61h5m15.109242178s) [upload=58574]: Next returned nil segment Jul 27 00:41:05.386699126 (61h5m15.109262156s) [upload=58574]: Waiting for error group managing segments... Jul 27 00:41:05.673420807 (61h5m15.395983847s) [noise=true , address=79.127.205.245:4242 , piece_id=GW3T62S6N6UHKIAEICIUTYDZD7SEPTYIIFGF2MCZZORKGJK26IKA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.751378121 (61h5m15.473941201s) [noise=true , address=79.127.205.235:4676 , piece_id=DL6OOBP52DR5H2MLV5JC6GVL2RTSMBBAPJJN2QCSWRWEF5ET3IPA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.785236732 (61h5m15.507799772s) [noise=true , address=79.127.205.233:4652 , piece_id=DQAKEWUAPGO7LIYGWB4GPNGRAXAQDAQEQFYIORV5VFPL6RNCDHTQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.963974118 (61h5m15.686537158s) [noise=true , address=64.49.13.26:4953 , piece_id=5Y4IGZPA6IVL3KBNBI4AMVRZ4LBMVNAMB6ZESZ5QUGUHUEAHSS4A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.245235561 (61h5m15.967798601s) [noise=true , address=77.90.31.18:4808 , piece_id=RZLEG2RVUXSMQTVXPN4B6NGDD5ZB7DWWTDIIQ4NKJJ2BOUQZ5S2A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.313042727 (61h5m16.035605767s) [noise=true , address=77.90.31.11:4562 , piece_id=CBZEYSFVV3US7ZO5UJQLBKRHUTX67IQEGOTMLFM2G2S5KOVF7RHA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.384514823 (61h5m16.107077863s) [noise=true , address=77.90.31.16:4746 , piece_id=5XL5RQKYILX3SST5ZRW6BOVMDPHBWIWK6D25HKMXBH2XQREKH3GQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:08.671465203 (61h5m18.394028243s) [noise=true , address=156.146.43.225:4449 , piece_id=CEP3OYAT54MORXPOFR7J7ZSZ2VVS4HX6EJXF7PAJ34ASULHAERUA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:16.014149055 (61h5m25.736712155s) [noise=true , address=79.127.201.215:4358 , piece_id=B5A2TJE7DNKIB3OOILCUGJ5R6MT4T34MYUDELYUVQQTIHGG4GNQQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:20.053999614 (61h5m29.776562654s) [noise=true , address=79.127.201.215:4356 , piece_id=J26JGG4UOVETDXUVYRJR742W7FYEPZS2G2PMF2OO5WWVFN75LPNQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:42:03.167926679 (61h6m12.890489719s) [noise=true , address=104.36.82.47:4533 , piece_id=AS5YO7FDM5LCMAK2JBJF6F5YJUPMS6E2P3U5TE5ATTETYJH3KMWA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:45696->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168008936 (61h6m12.890572006s) [noise=true , address=104.36.82.46:4511 , piece_id=VNUMKF4OMIJRIH4UP2URJPZ4I2I7YZ62RW6I4UYVO5Y54GRFCHVA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: write tcp 188.166.225.170:50204->104.36.82.46:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168075384 (61h6m12.890638444s) [noise=true , address=77.90.31.21:4906 , piece_id=EGX353TKU2ZDXQ4NBOINI3OP7FDJU4OFNHWV3NKFXSIKPJV5E4QA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 188.166.225.170:35840->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168094470 (61h6m12.89065752s) [noise=true , address=77.90.31.17:4798 , piece_id=NBI6QAZEMYHWCD5Q7PIHK3VKDFMCKZEG5CCCKQ2ZQEZ6ZHZIJ3RA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 188.166.225.170:58378->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168115911 (61h6m12.890678971s) [noise=true , address=104.36.82.50:4048 , piece_id=OCDELTG5CJVE5ZLFP73LIZCHOFOGFOFOVR3TQTXO5TI2DGXNU2KQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1rJd5FahWjKfvJstcoLE5uJGgtVj2uZirPpCJxbfGPGQ3j4qxC, address:104.36.82.50:4048): protocol: write tcp 188.166.225.170:35764->104.36.82.50:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168159725 (61h6m12.890722775s) [noise=true , address=104.36.82.53:4121 , piece_id=NJNJ4DJEZ54JW3SE74WOMPGSOTITSCF2CLWNB36MHWBSC63FMQGA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 188.166.225.170:58594->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168180704 (61h6m12.890743774s) [noise=true , address=104.36.82.12:4173 , piece_id=UUTD5HXRZRQ4EQRGAL7BFBDCFZCPJVJLNKL5LUPEOZ54XYBNSO3A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 188.166.225.170:59106->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168226292 (61h6m12.890789332s) [noise=true , address=104.36.82.54:4177 , piece_id=VUJ3AA7YIPUMKUFZBYVWGJQ5O3WKMQ5NZQGTHL4XSUSGZ7OVSVNQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:56836->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168255929 (61h6m12.890818969s) [seg_pos={0 0}, upload=58574]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:42:03.395637014 (61h6m13.118200054s) [upload=58574]: Done uploading segments... Jul 27 00:42:03.395661221 (61h6m13.118224261s) [upload=58574]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230167266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [X..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XX] duration: 30.238851395s FAIL (2025-07-27 00:41:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23890138s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XX.X.XX...X...............XX......X...XXX..XXX..X..X.X.....] duration: 34.303348428s FAIL (2025-07-27 00:41:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.303381652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X...X......X.XX...XXXX.....X..........X..X..X...XX.......] duration: 5.860691618s FAIL (2025-07-27 00:41:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.860713569s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.820388197s FAIL (2025-07-27 00:41:19 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.820413424s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [X.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..] duration: 6.038375126s FAIL (2025-07-27 00:41:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.038401729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.247202589s FAIL (2025-07-27 00:41:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 205.245:4224 , piece_id=UWWRGFFTGW745YAIWJPNWENQH7ODDNTLKV3T6MYIW7MGYLFET2QQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.778788856 (61h4m13.501351896s) [noise=true , address=45.91.116.8:4062 , piece_id=EZHA3GLP75N3K3USKVJM2DEA7SXDMXHVFGULJZO2JTUO4ZI36RIQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779031020 (61h4m13.50159407s) [noise=true , address=77.90.31.11:4574 , piece_id=WK62EN2C5XUPBOXRDYJMYSR3VIODNWCBLCBATNCEWFSP54CPNXUQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779093169 (61h4m13.501656209s) [noise=true , address=79.127.201.215:4359 , piece_id=TGE2EKKVJHHI2NYBUOREPEKAY2JLLYKYOXKTZKMIZ2KDAV3MF47A, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779266820 (61h4m13.501829871s) [noise=true , address=77.90.31.21:4909 , piece_id=W3JH3ATUSBJW7JP3S5HKWFZPD2P753J3F2RSQR6LMSRCZECJ7TFA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779346723 (61h4m13.501909784s) [noise=true , address=104.36.82.46:4499 , piece_id=VSCRLBZI4PP7C44BIVDEL5AQ4WQXPQJLZN5YCVGWD4OK7S2U6PQA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779592634 (61h4m13.502155684s) [noise=true , address=45.91.116.11:4130 , piece_id=OQCVPHLFS5TKB7LGNKAF55FMBEOUK5MO2NH5Q2M24HGD45I33JQQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779599256 (61h4m13.502162296s) [noise=true , address=79.127.219.38:4006 , piece_id=R6QVCRQA2HXCT3FJLAQUHDJ6JP3RLM3Q3NJYSK6OK5D54E6TZNCQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779912216 (61h4m13.502475276s) [noise=true , address=104.36.82.54:4177 , piece_id=JJDHXHGGORLE4D5S6GUTCVTKZJAVG72UQTEHUMVJFKJ6Z5DYAUVQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780018949 (61h4m13.502581989s) [noise=true , address=104.36.82.52:4085 , piece_id=S7LKZA4FYQZ5OF3GB2M3NMVCJGFDK47WFXMZ2G2WTV6ZHTJK3X7Q, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780189997 (61h4m13.502753037s) [noise=true , address=79.127.205.235:4688 , piece_id=IJPC7JNYBQDVL73WXLJDUTHHGPP64VLXA5NRHEBDPFBJ263PP6MQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780258408 (61h4m13.502821458s) [noise=true , address=79.127.205.234:4606 , piece_id=LZLYSPYE6G53UGKIJGV3CTMADCESKKAEVAB5AHKZRKXDXRH7IBEQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780439975 (61h4m13.503003015s) [noise=true , address=104.36.82.42:4823 , piece_id=XWJH5AOQKPHF6562VXXFD577IDTHGCBAKNFKVQDX6IXUQKMBFVYA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780510309 (61h4m13.503073359s) [noise=true , address=45.91.116.4:4964 , piece_id=MA3SA3FDE2TYN7NPTNYJP5ACFDPRQRTVTNFEP5AYV45F2YC7LG7A, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.781423437 (61h4m13.503986477s) [noise=true , address=104.36.82.47:4524 , piece_id=NUQXFYU5S6L6MUJ75MRLWY5B3QQOPKPEMFFSE4X65APN3HFSCYFQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.781667844 (61h4m13.504230894s) [noise=true , address=77.90.31.17:4784 , piece_id=UZMGJM6EXE63H5MVFGUKV2KDQ5UENNXBL7YVXBOSVP7VDZ4QK3YQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:04.880942902 (61h4m14.603505942s) [upload=58558]: Next returned nil segment Jul 27 00:40:04.880958422 (61h4m14.603521462s) [upload=58558]: Waiting for error group managing segments... Jul 27 00:40:05.172884399 (61h4m14.895447439s) [noise=true , address=79.127.205.234:4606 , piece_id=LZLYSPYE6G53UGKIJGV3CTMADCESKKAEVAB5AHKZRKXDXRH7IBEQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.326184366 (61h4m15.048747406s) [noise=true , address=79.127.205.245:4224 , piece_id=UWWRGFFTGW745YAIWJPNWENQH7ODDNTLKV3T6MYIW7MGYLFET2QQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.502534094 (61h4m15.225097144s) [noise=true , address=79.127.205.235:4688 , piece_id=IJPC7JNYBQDVL73WXLJDUTHHGPP64VLXA5NRHEBDPFBJ263PP6MQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.750199488 (61h4m15.472762528s) [noise=true , address=45.91.116.8:4062 , piece_id=EZHA3GLP75N3K3USKVJM2DEA7SXDMXHVFGULJZO2JTUO4ZI36RIQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.764593127 (61h4m15.487156167s) [noise=true , address=79.127.219.38:4006 , piece_id=R6QVCRQA2HXCT3FJLAQUHDJ6JP3RLM3Q3NJYSK6OK5D54E6TZNCQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:06.077637180 (61h4m15.80020023s) [noise=true , address=77.90.31.11:4574 , piece_id=WK62EN2C5XUPBOXRDYJMYSR3VIODNWCBLCBATNCEWFSP54CPNXUQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:10.015760555 (61h4m19.738323595s) [noise=true , address=45.91.116.4:4964 , piece_id=MA3SA3FDE2TYN7NPTNYJP5ACFDPRQRTVTNFEP5AYV45F2YC7LG7A, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:15.262299954 (61h4m24.984862984s) [noise=true , address=45.91.116.9:4277 , piece_id=LI6H7STY7PC6KKWC346EB6342YR56FSBELYTGY4ZL4Y7IS5GVYCA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:40:17.833459686 (61h4m27.556022726s) [noise=true , address=45.91.116.11:4130 , piece_id=OQCVPHLFS5TKB7LGNKAF55FMBEOUK5MO2NH5Q2M24HGD45I33JQQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:37.513430104 (61h4m47.235993144s) [noise=true , address=79.127.201.215:4359 , piece_id=TGE2EKKVJHHI2NYBUOREPEKAY2JLLYKYOXKTZKMIZ2KDAV3MF47A, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:41:02.672822384 (61h5m12.395385424s) [noise=true , address=104.36.82.52:4085 , piece_id=S7LKZA4FYQZ5OF3GB2M3NMVCJGFDK47WFXMZ2G2WTV6ZHTJK3X7Q, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:35886->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672847302 (61h5m12.395410342s) [noise=true , address=104.36.82.46:4499 , piece_id=VSCRLBZI4PP7C44BIVDEL5AQ4WQXPQJLZN5YCVGWD4OK7S2U6PQA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:39188->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672880225 (61h5m12.395443265s) [noise=true , address=104.36.82.47:4524 , piece_id=NUQXFYU5S6L6MUJ75MRLWY5B3QQOPKPEMFFSE4X65APN3HFSCYFQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672902898 (61h5m12.395465938s) [noise=true , address=104.36.82.42:4823 , piece_id=XWJH5AOQKPHF6562VXXFD577IDTHGCBAKNFKVQDX6IXUQKMBFVYA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: rpc: dial tcp 104.36.82.42:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672924129 (61h5m12.395487169s) [noise=true , address=104.36.82.54:4177 , piece_id=JJDHXHGGORLE4D5S6GUTCVTKZJAVG72UQTEHUMVJFKJ6Z5DYAUVQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:42474->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672957903 (61h5m12.395520943s) [noise=true , address=104.36.82.14:4233 , piece_id=73MXEVAOSLY2GXT4PIPH2EPRZLLAFZYLN3ZMEBZQQMXUWQKPIPHQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:124wXnmRuxo3o7AVwdSbUmai3SebaMV5qgScGVFSWFKSRvBNTyB, address:104.36.82.14:4233): protocol: write tcp 188.166.225.170:57380->104.36.82.14:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672961260 (61h5m12.3955243s) [noise=true , address=77.90.31.17:4784 , piece_id=UZMGJM6EXE63H5MVFGUKV2KDQ5UENNXBL7YVXBOSVP7VDZ4QK3YQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection; write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection Jul 27 00:41:02.672997288 (61h5m12.395560328s) [noise=true , address=77.90.31.21:4909 , piece_id=W3JH3ATUSBJW7JP3S5HKWFZPD2P753J3F2RSQR6LMSRCZECJ7TFA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 188.166.225.170:52332->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.673020443 (61h5m12.395583483s) [seg_pos={0 0}, upload=58558]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:41:02.918063519 (61h5m12.640626559s) [upload=58558]: Done uploading segments... Jul 27 00:41:02.918091382 (61h5m12.640654422s) [upload=58558]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.247215343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX......] duration: 7.61103776s FAIL (2025-07-27 00:40:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.611245738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX.] duration: 7.318737466s FAIL (2025-07-27 00:40:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.318771231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..X] duration: 30.233844013s FAIL (2025-07-27 00:40:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.233898618s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X....] duration: 32.695061306s FAIL (2025-07-27 00:40:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.695080031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.830264335s FAIL (2025-07-27 00:40:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.830285996s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX] duration: 30.231884295s FAIL (2025-07-27 00:40:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.231908392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.240808854s FAIL (2025-07-27 00:40:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 96 (61h3m13.006453136s) [seg_pos={0 0}, upload=58546]: Begin upload segment... Jul 27 00:39:03.283954238 (61h3m13.006517288s) [seg_pos={0 0}, upload=58546]: Done begin upload segment. Jul 27 00:39:03.283964557 (61h3m13.006527678s) [noise=true , address=79.127.205.234:4612 , piece_id=7EJPSP6UGSBXD2L7IO5NUDSLZ3Q4L6VWCL3HH2UX2JZ3MWHGPMXA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.283976491 (61h3m13.006539531s) [noise=true , address=64.49.13.140:4508 , piece_id=5NJSO6573FLHESKZ7FBEOJWISL4VCIKQEPTVCLV2CFW355JUJBWA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284391785 (61h3m13.006954825s) [noise=true , address=104.36.82.47:4548 , piece_id=TTHWZQZJNYW37MNGI2TLXI3HHIJPLECEV72CETXWLU3XJAZSANHA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284689966 (61h3m13.007253006s) [noise=true , address=104.36.82.56:4247 , piece_id=JSZJHOX2FNB6MINTYPEEWGB2SISWZZHBU3FMRUY4HXJDD7L5GN6Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284897492 (61h3m13.007460542s) [noise=true , address=64.49.13.138:4482 , piece_id=7O5XVSB5TRM4KRLYDP3P5E52VWRERWKMAYPK4DQBLTZUVXSX466Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285010729 (61h3m13.007573769s) [noise=true , address=77.90.31.12:4597 , piece_id=LAAI66KTPINHH4QQVVWTH56KM4UHG7WZCY5SEWZLDSLPQGYIPPXA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285174092 (61h3m13.007737122s) [noise=true , address=104.36.82.46:4489 , piece_id=LVIKRAOP2PE3FZPMQPUKBGK5OS4AQSDEHKN3LDMT7GOPVNBIGACA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285316995 (61h3m13.007880035s) [noise=true , address=104.36.82.37:4685 , piece_id=QXVTM2B6RG6HBSDBBEGBRQXKNKPTII37RJM2OXLQDOFEJ3H7WUFA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285555291 (61h3m13.008118331s) [noise=true , address=77.90.31.18:4833 , piece_id=FNSNVHPXYAKYY7VMER4V55Q7W3NT4IDCRR3537DLJEBVRIHJII6Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285921642 (61h3m13.008484682s) [noise=true , address=79.127.205.235:4669 , piece_id=HR37H5QGSOIM3DGGOBTNUG3VVBR3JQLMV46VPPUGWK7FCYYOKKGQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286310306 (61h3m13.008873346s) [noise=true , address=79.127.205.233:4624 , piece_id=6T4S7Q2FGJNULWMYK3BD244FW24EKICU2CKSE3KRR6AJNGVTRSTQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286476755 (61h3m13.009039805s) [noise=true , address=77.90.31.11:4565 , piece_id=HF4PPC2URIELIUOFOPLVJHYDNQ5ZMSV3OD56XKL62GAAK2N7T4RQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286813248 (61h3m13.009376298s) [noise=true , address=104.36.82.54:4179 , piece_id=GSSXZ5XTWYDYM43WNGC3OBF7HIWTB7AANQBTF3SAL44VCP67HIMA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287136868 (61h3m13.009699908s) [noise=true , address=104.36.82.13:4217 , piece_id=MPLNAZQJ2FSLK65JCTZZ5BXWBLUTJIJR7LAJSMBGVAWLCCWKZS4A, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287234844 (61h3m13.009797885s) [noise=true , address=104.36.82.52:4093 , piece_id=WU2NHJSSH5HMKGKR3ZPS2O7L7BHGOLRXVO4NY5NAMDEY2KHX3KOA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287586958 (61h3m13.010150008s) [noise=true , address=79.127.205.246:4277 , piece_id=DIM3L5H663A6N6D6QMQQGHDGYGWVDP4VB5C43JDUJRFVJ2IH2ELA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287922600 (61h3m13.01048564s) [noise=true , address=79.127.220.101:4041 , piece_id=KDCVMSDLMI7MLBE2O5IZTDKYLC6QWJMPCOC32U4X2QA2XXKWOJTQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.288278451 (61h3m13.010841491s) [noise=true , address=45.91.116.14:4206 , piece_id=W43PZ4MVKZCEIBE55Z6I2FGDAG7G5T2DQO3UYO7WSOIRAPGRTLVQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:04.388954830 (61h3m14.11151788s) [upload=58546]: Next returned nil segment Jul 27 00:39:04.388981491 (61h3m14.111544531s) [upload=58546]: Waiting for error group managing segments... Jul 27 00:39:04.685103236 (61h3m14.407666277s) [noise=true , address=79.127.205.234:4612 , piece_id=7EJPSP6UGSBXD2L7IO5NUDSLZ3Q4L6VWCL3HH2UX2JZ3MWHGPMXA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.689990508 (61h3m14.412553548s) [noise=true , address=79.127.205.235:4669 , piece_id=HR37H5QGSOIM3DGGOBTNUG3VVBR3JQLMV46VPPUGWK7FCYYOKKGQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.692098300 (61h3m14.41466134s) [noise=true , address=79.127.205.246:4277 , piece_id=DIM3L5H663A6N6D6QMQQGHDGYGWVDP4VB5C43JDUJRFVJ2IH2ELA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.706594225 (61h3m14.429157275s) [noise=true , address=79.127.205.233:4624 , piece_id=6T4S7Q2FGJNULWMYK3BD244FW24EKICU2CKSE3KRR6AJNGVTRSTQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.101755451 (61h3m14.824318491s) [noise=true , address=64.49.13.140:4508 , piece_id=5NJSO6573FLHESKZ7FBEOJWISL4VCIKQEPTVCLV2CFW355JUJBWA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.304648493 (61h3m15.027211523s) [noise=true , address=77.90.31.18:4833 , piece_id=FNSNVHPXYAKYY7VMER4V55Q7W3NT4IDCRR3537DLJEBVRIHJII6Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.347548484 (61h3m15.070111524s) [noise=true , address=77.90.31.11:4565 , piece_id=HF4PPC2URIELIUOFOPLVJHYDNQ5ZMSV3OD56XKL62GAAK2N7T4RQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.408810325 (61h3m15.131373365s) [noise=true , address=45.91.116.14:4206 , piece_id=W43PZ4MVKZCEIBE55Z6I2FGDAG7G5T2DQO3UYO7WSOIRAPGRTLVQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:07.062675451 (61h3m16.785238491s) [noise=true , address=77.90.31.12:4597 , piece_id=LAAI66KTPINHH4QQVVWTH56KM4UHG7WZCY5SEWZLDSLPQGYIPPXA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:07.433784748 (61h3m17.156347788s) [noise=true , address=79.127.220.101:4041 , piece_id=KDCVMSDLMI7MLBE2O5IZTDKYLC6QWJMPCOC32U4X2QA2XXKWOJTQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:12.527684124 (61h3m22.250247154s) [noise=true , address=64.49.13.138:4482 , piece_id=7O5XVSB5TRM4KRLYDP3P5E52VWRERWKMAYPK4DQBLTZUVXSX466Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:40:02.190434180 (61h4m11.91299723s) [noise=true , address=104.36.82.47:4548 , piece_id=TTHWZQZJNYW37MNGI2TLXI3HHIJPLECEV72CETXWLU3XJAZSANHA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190541675 (61h4m11.913104726s) [noise=true , address=104.36.82.56:4247 , piece_id=JSZJHOX2FNB6MINTYPEEWGB2SISWZZHBU3FMRUY4HXJDD7L5GN6Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190605237 (61h4m11.913168287s) [noise=true , address=104.36.82.54:4179 , piece_id=GSSXZ5XTWYDYM43WNGC3OBF7HIWTB7AANQBTF3SAL44VCP67HIMA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: write tcp 188.166.225.170:53772->104.36.82.54:4179: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190653319 (61h4m11.913216359s) [noise=true , address=104.36.82.46:4489 , piece_id=LVIKRAOP2PE3FZPMQPUKBGK5OS4AQSDEHKN3LDMT7GOPVNBIGACA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190669720 (61h4m11.91323277s) [noise=true , address=104.36.82.37:4685 , piece_id=QXVTM2B6RG6HBSDBBEGBRQXKNKPTII37RJM2OXLQDOFEJ3H7WUFA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM, address:104.36.82.37:4685): protocol: rpc: dial tcp 104.36.82.37:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190718413 (61h4m11.913281453s) [noise=true , address=104.36.82.13:4217 , piece_id=MPLNAZQJ2FSLK65JCTZZ5BXWBLUTJIJR7LAJSMBGVAWLCCWKZS4A, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:104.36.82.13:4217): protocol: rpc: dial tcp 104.36.82.13:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190768619 (61h4m11.913331669s) [noise=true , address=104.36.82.52:4093 , piece_id=WU2NHJSSH5HMKGKR3ZPS2O7L7BHGOLRXVO4NY5NAMDEY2KHX3KOA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 188.166.225.170:32968->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190797224 (61h4m11.913360535s) [seg_pos={0 0}, upload=58546]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:40:02.429216719 (61h4m12.151779769s) [upload=58546]: Done uploading segments... Jul 27 00:40:02.429248249 (61h4m12.151811289s) [upload=58546]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.240825697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X...] duration: 34.184643634s FAIL (2025-07-27 00:39:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.184654905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..] duration: 7.911199827s FAIL (2025-07-27 00:39:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.911219945s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX.....] duration: 6.504881858s FAIL (2025-07-27 00:39:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.504907338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXX] duration: 30.235322881s FAIL (2025-07-27 00:39:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tore close: conn closed: op Jul 27 00:38:26.756503444 (61h2m36.479066494s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:35416->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756520356 (61h2m36.479083386s) [seg_pos={0 0}, upload=58529]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:38:26.982864478 (61h2m36.705427518s) [upload=58529]: Done uploading segments... Jul 27 00:38:26.982888764 (61h2m36.705451804s) [upload=58529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:39:00.518757775 (61h3m10.241320815s) [upload=58544]: Starting upload Jul 27 00:39:00.518861763 (61h3m10.241424813s) [upload=58544]: Done starting upload Jul 27 00:39:00.518978487 (61h3m10.241541527s) [upload=58544]: Uploading segments... Jul 27 00:39:00.519190152 (61h3m10.241753192s) [upload=58544]: Got next segment. Inline: false Jul 27 00:39:00.519268151 (61h3m10.241831201s) [upload=58544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:00.519274303 (61h3m10.241837363s) [upload=58544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:00.772196596 (61h3m10.494759646s) [seg_pos={0 0}, upload=58544]: Begin upload segment... Jul 27 00:39:00.772257002 (61h3m10.494820042s) [seg_pos={0 0}, upload=58544]: Done begin upload segment. Jul 27 00:39:00.772278513 (61h3m10.494841553s) [noise=true , address=77.90.31.12:4590 , piece_id=TDK7WYMWOSNAEQW55ENFSZY5P6UCAUCF7BW6SZ26UCFDI3KM2H7Q, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772292560 (61h3m10.4948556s) [noise=true , address=45.91.116.13:4154 , piece_id=ACG7XISQPIIM7BZWJ2FXZAM577TXZNIRRIAKKAEWVFYHIWTSLQ7Q, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772350882 (61h3m10.494913922s) [noise=true , address=45.91.116.6:4973 , piece_id=LR4FNOBTP2GN2FAFPJIKONKH32CHK2UWZ4XQFJ5NAHNCAZLZBSJA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772823256 (61h3m10.495386296s) [noise=true , address=45.91.116.9:4263 , piece_id=BCHXOV7YM7GNNWTJXGQNQ56GHTNAJQ6XL3APJFYNAVND4OZTX3HQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772871358 (61h3m10.495434388s) [noise=true , address=104.36.82.4:4804 , piece_id=LSZ2OOIYJQPQSXE34PHR3FA5EZW7CZ6HQJYPKYBN3CQLKJEU4DLQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773245774 (61h3m10.495808824s) [noise=true , address=45.91.116.9:4262 , piece_id=5LBV4SRCIDQYFC6LPF2ZEBK2GJODZ57YCYHIFZSFK3EDTSE6LHBA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773337259 (61h3m10.495900299s) [noise=true , address=77.90.31.14:4663 , piece_id=RW6KFZSSRJDGEDYWEVXDQUBHVRQTVUWBNNSHJGKELNNZ2H4XOZQA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773618197 (61h3m10.496181237s) [noise=true , address=77.90.31.19:4843 , piece_id=FRDM2XKZ3RWV224KIACONFUZ2TLULLWUM25Q5EI4NMCLO7R62LOA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773695344 (61h3m10.496258384s) [noise=true , address=45.91.116.4:4938 , piece_id=KPM5PXZK4TKFYZ24ATHXIJQWTHJYHIQMNSHZGJ2SGXW7A657PYKA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774118363 (61h3m10.496681413s) [noise=true , address=45.91.116.14:4190 , piece_id=SL5CNE3TFOIX5TMHYB6ZF7LT3PBKZWCUQ7NOKRHO6FX4EATXIRZA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774497489 (61h3m10.497060529s) [noise=true , address=104.36.82.16:4293 , piece_id=5L3LGH75ABJLM6LG2NZYDIJQGRYNJ5AN2I3FDXO64FRYIMLVFBEA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774897784 (61h3m10.497460824s) [noise=true , address=77.90.31.13:4626 , piece_id=SF5O3K3ZNRCBOIW5A3K6GCBPDXVQJ6NBY57GXFJSY6BNHDVZ7ACA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.775283312 (61h3m10.497846352s) [noise=true , address=45.91.116.10:4082 , piece_id=RUUBCAPVSVBUZLL3GIFJLV4I52SLZJNAZNUXQU3V3KMKX5YKILUA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.775643621 (61h3m10.498206661s) [noise=true , address=77.90.31.21:4879 , piece_id=N3HYMQAOXGHJRLR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.776021434 (61h3m10.498584484s) [noise=true , address=77.90.31.15:4699 , piece_id=PBREYVQWOB54Q7VQYLEF3FPDN6W4GX4L2BT6F4UOLVPXVD4KEIYQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.776339173 (61h3m10.498902213s) [noise=true , address=45.91.116.11:4118 , piece_id=WNW4NVJMKKGCOPD5EHV34LJ5MACPSTVS42C4KEA6VEXG6IAINZLA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:02.523350266 (61h3m12.245913306s) [upload=58544]: Next returned nil segment Jul 27 00:39:02.523372387 (61h3m12.245935427s) [upload=58544]: Waiting for error group managing segments... Jul 27 00:39:02.803441189 (61h3m12.526004229s) [noise=true , address=77.90.31.14:4663 , piece_id=RW6KFZSSRJDGEDYWEVXDQUBHVRQTVUWBNNSHJGKELNNZ2H4XOZQA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.840550059 (61h3m12.563113099s) [noise=true , address=45.91.116.14:4190 , piece_id=SL5CNE3TFOIX5TMHYB6ZF7LT3PBKZWCUQ7NOKRHO6FX4EATXIRZA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.869696104 (61h3m12.592259144s) [noise=true , address=45.91.116.11:4118 , piece_id=WNW4NVJMKKGCOPD5EHV34LJ5MACPSTVS42C4KEA6VEXG6IAINZLA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.879743457 (61h3m12.602306497s) [noise=true , address=45.91.116.13:4154 , piece_id=ACG7XISQPIIM7BZWJ2FXZAM577TXZNIRRIAKKAEWVFYHIWTSLQ7Q, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.885416381 (61h3m12.607979421s) [noise=true , address=45.91.116.10:4082 , piece_id=RUUBCAPVSVBUZLL3GIFJLV4I52SLZJNAZNUXQU3V3KMKX5YKILUA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.888715092 (61h3m12.611278132s) [noise=true , address=45.91.116.6:4973 , piece_id=LR4FNOBTP2GN2FAFPJIKONKH32CHK2UWZ4XQFJ5NAHNCAZLZBSJA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.898264905 (61h3m12.620827945s) [noise=true , address=45.91.116.4:4938 , piece_id=KPM5PXZK4TKFYZ24ATHXIJQWTHJYHIQMNSHZGJ2SGXW7A657PYKA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.956219137 (61h3m12.678782177s) [noise=true , address=45.91.116.9:4262 , piece_id=5LBV4SRCIDQYFC6LPF2ZEBK2GJODZ57YCYHIFZSFK3EDTSE6LHBA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.956593604 (61h3m12.679156644s) [noise=true , address=45.91.116.9:4263 , piece_id=BCHXOV7YM7GNNWTJXGQNQ56GHTNAJQ6XL3APJFYNAVND4OZTX3HQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:04.245404156 (61h3m13.967967197s) [noise=true , address=77.90.31.19:4843 , piece_id=FRDM2XKZ3RWV224KIACONFUZ2TLULLWUM25Q5EI4NMCLO7R62LOA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:04.249864841 (61h3m13.972427881s) [noise=true , address=77.90.31.13:4626 , piece_id=SF5O3K3ZNRCBOIW5A3K6GCBPDXVQJ6NBY57GXFJSY6BNHDVZ7ACA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:05.008337626 (61h3m14.730900666s) [noise=true , address=77.90.31.12:4590 , piece_id=TDK7WYMWOSNAEQW55ENFSZY5P6UCAUCF7BW6SZ26UCFDI3KM2H7Q, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:29.867376960 (61h3m39.58994s) [noise=true , address=104.36.82.4:4804 , piece_id=LSZ2OOIYJQPQSXE34PHR3FA5EZW7CZ6HQJYPKYBN3CQLKJEU4DLQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:47338->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867392059 (61h3m39.589955099s) [noise=true , address=77.90.31.15:4699 , piece_id=PBREYVQWOB54Q7VQYLEF3FPDN6W4GX4L2BT6F4UOLVPXVD4KEIYQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867398612 (61h3m39.589961652s) [noise=true , address=104.36.82.16:4293 , piece_id=5L3LGH75ABJLM6LG2NZYDIJQGRYNJ5AN2I3FDXO64FRYIMLVFBEA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:49496->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867422488 (61h3m39.589985528s) [noise=true , address=77.90.31.21:4879 , piece_id=N3HYMQAOXGHJRLR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235345144s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX....] duration: 6.349223489s FAIL (2025-07-27 00:39:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.349242905s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X..] duration: 49.999475584s FAIL (2025-07-27 00:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.999486144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.842763924s FAIL (2025-07-27 00:39:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.842795825s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX.] duration: 7.60010323s FAIL (2025-07-27 00:39:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.600132556s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238455521s FAIL (2025-07-27 00:39:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ess=104.36.82.56:4232 , piece_id=BBGYLAUDFKIMNRMUFAP53WKAR2KKTNFEM33NMV6C34Q6WV7RKKTQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.813599578 (61h2m12.536162618s) [noise=true , address=64.49.13.139:4478 , piece_id=OA2RYNGH4GL43RTSIPYY5NVSGIWV7JLRTENC4BQMBYY2MOI5XYPA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814356225 (61h2m12.536919255s) [noise=true , address=104.36.82.54:4180 , piece_id=X43ASH4NVEFA5E7H7WIHIVOCJRNYLEOUZ6SGXA3OGZ2HKMUX722Q, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814637593 (61h2m12.537200633s) [noise=true , address=77.90.31.15:4727 , piece_id=JGQA7LS5QWQZFKAU7VEICWHX643IALQFAKXH3D3NHYG3WSTG2ZHA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814940092 (61h2m12.537503132s) [noise=true , address=104.36.82.53:4121 , piece_id=Z3GZITDFNCZXGLYPV2ASPVNRSSWHP5PMKR55R7W4DLYMKB6P5FBQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.815231099 (61h2m12.537794139s) [noise=true , address=64.49.13.138:4482 , piece_id=AMQTARPP6LQ3RGFJWA2NZD25KNRUL2UVZ35SBJB3GSAVOF5D3LSQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.815514090 (61h2m12.53807712s) [noise=true , address=77.90.31.11:4574 , piece_id=MMNHJYAYE57UQRMOKKIL5JJOJHJMHSGZZO5XMXCSQCRTANFJX2YQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.816467815 (61h2m12.539030865s) [noise=true , address=104.36.82.13:4209 , piece_id=POVGOCUKLKHVDYTDICHN34JK7UUVXBBGXI6YKPALAB6WWAE4YZDA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.816825570 (61h2m12.53938862s) [noise=true , address=79.127.205.234:4610 , piece_id=H7BKX2PCNY4LWW4GIXP3N2VN4FT3IBCMANFWIQBQJQKVZLXXYPRA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817121366 (61h2m12.539684396s) [noise=true , address=77.90.31.10:4534 , piece_id=7T4PYLVXFNYMKD6WUU2FBOE2WOLYJVBMHXA7J2MAC6SFFQGYXUSA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817560326 (61h2m12.540123366s) [noise=true , address=79.127.205.246:4262 , piece_id=JVPKREGGUI7NAORUZERP6ZCSZ5YO5K5KR54X52WA4TLATNDXK6YA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817926045 (61h2m12.540489085s) [noise=true , address=79.127.205.233:4655 , piece_id=AUDCFQYWG6V77VLICZ4CWCU3G3PX6FSDPR3OS4U5E4EXAI5DJ4MA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.818258391 (61h2m12.540821441s) [noise=true , address=104.36.82.46:4516 , piece_id=VL3UZT2SBIDBF74L6F2BA3LXQQNSPSQN7XXUCB5FSDGNJVVURCSQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.818839833 (61h2m12.541402873s) [noise=true , address=79.127.205.245:4242 , piece_id=Z2V4VN3EMYWYZPGKUMHJ4YNXM3DAGDUPSMEYQD74ZIUARIUZHQ5A, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.819111042 (61h2m12.541674082s) [noise=true , address=77.90.31.13:4634 , piece_id=KF4OEJ42UQ56MIYSJVE47DBFQDY2ZED3D5D22CGCZP4O67W7SGZA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.819444530 (61h2m12.54200757s) [noise=true , address=79.127.219.38:4001 , piece_id=LF2H5R52QMJGXXA6ZBGPQS2RRKPIDEUDRPSDLMXYSGMBD32UMD4A, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:03.892037167 (61h2m13.614600207s) [noise=true , address=77.90.31.13:4634 , piece_id=KF4OEJ42UQ56MIYSJVE47DBFQDY2ZED3D5D22CGCZP4O67W7SGZA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:03.912708226 (61h2m13.635271266s) [upload=58531]: Next returned nil segment Jul 27 00:38:03.912726081 (61h2m13.635289111s) [upload=58531]: Waiting for error group managing segments... Jul 27 00:38:04.203079740 (61h2m13.92564278s) [noise=true , address=79.127.205.234:4610 , piece_id=H7BKX2PCNY4LWW4GIXP3N2VN4FT3IBCMANFWIQBQJQKVZLXXYPRA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.263993836 (61h2m13.986556876s) [noise=true , address=79.127.205.245:4242 , piece_id=Z2V4VN3EMYWYZPGKUMHJ4YNXM3DAGDUPSMEYQD74ZIUARIUZHQ5A, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.277880706 (61h2m14.000443746s) [noise=true , address=79.127.205.246:4262 , piece_id=JVPKREGGUI7NAORUZERP6ZCSZ5YO5K5KR54X52WA4TLATNDXK6YA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.593509724 (61h2m14.316072754s) [noise=true , address=79.127.205.233:4655 , piece_id=AUDCFQYWG6V77VLICZ4CWCU3G3PX6FSDPR3OS4U5E4EXAI5DJ4MA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.708002823 (61h2m14.430565853s) [noise=true , address=79.127.219.38:4001 , piece_id=LF2H5R52QMJGXXA6ZBGPQS2RRKPIDEUDRPSDLMXYSGMBD32UMD4A, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.801373607 (61h2m14.523936647s) [noise=true , address=77.90.31.11:4574 , piece_id=MMNHJYAYE57UQRMOKKIL5JJOJHJMHSGZZO5XMXCSQCRTANFJX2YQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:05.697093515 (61h2m15.419656555s) [noise=true , address=64.49.13.138:4482 , piece_id=AMQTARPP6LQ3RGFJWA2NZD25KNRUL2UVZ35SBJB3GSAVOF5D3LSQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:18.733661915 (61h2m28.456224995s) [noise=true , address=64.49.13.139:4478 , piece_id=OA2RYNGH4GL43RTSIPYY5NVSGIWV7JLRTENC4BQMBYY2MOI5XYPA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:39:01.728045200 (61h3m11.45060825s) [noise=true , address=77.90.31.10:4534 , piece_id=7T4PYLVXFNYMKD6WUU2FBOE2WOLYJVBMHXA7J2MAC6SFFQGYXUSA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:01.728566166 (61h3m11.451129206s) [noise=true , address=104.36.82.56:4232 , piece_id=BBGYLAUDFKIMNRMUFAP53WKAR2KKTNFEM33NMV6C34Q6WV7RKKTQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728607806 (61h3m11.451170856s) [noise=true , address=77.90.31.15:4727 , piece_id=JGQA7LS5QWQZFKAU7VEICWHX643IALQFAKXH3D3NHYG3WSTG2ZHA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728643524 (61h3m11.451206564s) [noise=true , address=104.36.82.53:4121 , piece_id=Z3GZITDFNCZXGLYPV2ASPVNRSSWHP5PMKR55R7W4DLYMKB6P5FBQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728688470 (61h3m11.45125152s) [noise=true , address=104.36.82.46:4516 , piece_id=VL3UZT2SBIDBF74L6F2BA3LXQQNSPSQN7XXUCB5FSDGNJVVURCSQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: rpc: dial tcp 104.36.82.46:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728723847 (61h3m11.451286887s) [noise=true , address=104.36.82.47:4520 , piece_id=DHTRSISHBBTWDK2DTURXLPFSXBZECIR24AVBW5X5YANNBJWJIZ7Q, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: rpc: dial tcp 104.36.82.47:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728760208 (61h3m11.451323248s) [noise=true , address=104.36.82.13:4209 , piece_id=POVGOCUKLKHVDYTDICHN34JK7UUVXBBGXI6YKPALAB6WWAE4YZDA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:18W2j3VhAMCmWAuST1M1yLgCWBZP3hQwGkLUvFAMSzUmhLpVRc, address:104.36.82.13:4209): protocol: rpc: dial tcp 104.36.82.13:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728801165 (61h3m11.451364215s) [noise=true , address=104.36.82.54:4180 , piece_id=X43ASH4NVEFA5E7H7WIHIVOCJRNYLEOUZ6SGXA3OGZ2HKMUX722Q, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728841623 (61h3m11.451404663s) [noise=true , address=104.36.82.9:4039 , piece_id=6LQJYVPLRYHPY6ZCNNHYDMPBCGBSTGKK3QKS6M4IK7MFUWADCIFQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:4039): protocol: write tcp 188.166.225.170:52344->104.36.82.9:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728858355 (61h3m11.451421395s) [seg_pos={0 0}, upload=58531]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:39:01.963894737 (61h3m11.686457777s) [upload=58531]: Done uploading segments... Jul 27 00:39:01.963933591 (61h3m11.686496631s) [upload=58531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.238495137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX] duration: 30.231335524s FAIL (2025-07-27 00:38:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23135541s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.851054536s FAIL (2025-07-27 00:38:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.85108238s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X.] duration: 55.559956416s FAIL (2025-07-27 00:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.559975772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXX] duration: 30.227796764s FAIL (2025-07-27 00:38:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 27 00:37:25.753540791 (61h1m35.476103861s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; context deadline exceeded Jul 27 00:37:25.753560599 (61h1m35.476123629s) [seg_pos={0 0}, upload=58512]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:37:25.982334529 (61h1m35.704897569s) [upload=58512]: Done uploading segments... Jul 27 00:37:25.982363554 (61h1m35.704926594s) [upload=58512]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:37:57.395119345 (61h2m7.117682375s) [upload=58529]: Starting upload Jul 27 00:37:57.395186974 (61h2m7.117750004s) [upload=58529]: Done starting upload Jul 27 00:37:57.395260806 (61h2m7.117823836s) [upload=58529]: Uploading segments... Jul 27 00:37:57.395377409 (61h2m7.117940449s) [upload=58529]: Got next segment. Inline: false Jul 27 00:37:57.395415763 (61h2m7.117978803s) [upload=58529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:37:57.395420442 (61h2m7.117983482s) [upload=58529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:37:57.838860108 (61h2m7.561423138s) [seg_pos={0 0}, upload=58529]: Begin upload segment... Jul 27 00:37:57.838903682 (61h2m7.561466712s) [seg_pos={0 0}, upload=58529]: Done begin upload segment. Jul 27 00:37:57.838926074 (61h2m7.561489114s) [noise=true , address=104.36.82.13:4186 , piece_id=FOWJ63ZEQLGVLC5UCVPHU2DNGY2POPR2XSH4VK6KORUWRFZNRHYA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839032508 (61h2m7.561595538s) [noise=true , address=45.91.116.5:4299 , piece_id=N3JVJ3VWTNEEML7PPGERQ632YVEY6R5DVCPJ7YOPCKJO5EEQLLGQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839290873 (61h2m7.561853913s) [noise=true , address=45.91.116.12:4225 , piece_id=7W5TMYBFLWMAHHCTZPZECF4KXLQ7M246ZVSTQ2FJNFZ4OWLF2HTQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839504170 (61h2m7.56206721s) [noise=true , address=45.91.116.9:4263 , piece_id=NRLM5DFHZL5RZM2DUIYXKDLOIMIBIZKPSXAUG3FCDYTCA4CF54WA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839588593 (61h2m7.562151633s) [noise=true , address=77.90.31.15:4698 , piece_id=Y4YNEMHV3VTRZUUZFSTZTHC2HIMFPYUTPYLRYZ6NCGOI34NQXPDQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839784357 (61h2m7.562347397s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839928182 (61h2m7.562491212s) [noise=true , address=45.91.116.13:4153 , piece_id=3CJCYVMEM24WNJFMHEWPQB3HVPB7HWJQ22XTUFXJNCO7T7LBRPSQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840119398 (61h2m7.562682438s) [noise=true , address=45.91.116.10:4081 , piece_id=QXFA3TKUKIURB2POZVRMACPUFBT4GHJLLPLCCKEVGDZDZKOCFIBQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840189011 (61h2m7.562752051s) [noise=true , address=45.91.116.8:4045 , piece_id=TFKIFVA6FMBE63ILNB6QTJBHPLYL7PEKSTGZ3LWRFUST2RMNFPXA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840414092 (61h2m7.562977132s) [noise=true , address=77.90.31.11:4553 , piece_id=E4MGVVEF3CX6RCKIFMZ7GAMHHPJAPANEFSGQKEJ4SQW4IYKE7FOA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840429411 (61h2m7.562992451s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXDQGDHP4NI633USSPA2NW4SBKW4OGUG764B5EXA6AKUULYLEAQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840696472 (61h2m7.563259512s) [noise=true , address=77.90.31.12:4590 , piece_id=ZCZF62HABMW4EVT4OBMWCPFAFMHY6LPYM7QJVFOBAH2HHIZT2UMA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840701151 (61h2m7.563264191s) [noise=true , address=77.90.31.14:4663 , piece_id=KE3ZNOFT3YI7FOVFPNHNBAMSF7NMUFYR23QEU7TXZNURLBD3MEFQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840940930 (61h2m7.56350397s) [noise=true , address=45.91.116.4:4939 , piece_id=YTQBXZKLMAB7KGB7BMYM7TA6J4NVMU6N7GUTAIK77DLRGY2D2A6A, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840974193 (61h2m7.563537223s) [noise=true , address=45.91.116.11:4117 , piece_id=ZPVFASEDZOJ7PH2NOB3S4NIRH5TDW7OGTJUOLCQCZAAPGKUB35LA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.841186480 (61h2m7.56374952s) [noise=true , address=104.36.82.2:4732 , piece_id=O5UZK5W5HRROTFV45TPI35OLW2XLSVO4PPVCHQZVCNJYTMZ4YFDA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:59.572735848 (61h2m9.295298888s) [upload=58529]: Next returned nil segment Jul 27 00:37:59.572757530 (61h2m9.29532056s) [upload=58529]: Waiting for error group managing segments... Jul 27 00:37:59.849447747 (61h2m9.572010787s) [noise=true , address=45.91.116.13:4153 , piece_id=3CJCYVMEM24WNJFMHEWPQB3HVPB7HWJQ22XTUFXJNCO7T7LBRPSQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.849732161 (61h2m9.572295191s) [noise=true , address=77.90.31.14:4663 , piece_id=KE3ZNOFT3YI7FOVFPNHNBAMSF7NMUFYR23QEU7TXZNURLBD3MEFQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.855079944 (61h2m9.577642984s) [noise=true , address=45.91.116.12:4225 , piece_id=7W5TMYBFLWMAHHCTZPZECF4KXLQ7M246ZVSTQ2FJNFZ4OWLF2HTQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.934575471 (61h2m9.657138511s) [noise=true , address=45.91.116.10:4081 , piece_id=QXFA3TKUKIURB2POZVRMACPUFBT4GHJLLPLCCKEVGDZDZKOCFIBQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.937599247 (61h2m9.660162277s) [noise=true , address=77.90.31.11:4553 , piece_id=E4MGVVEF3CX6RCKIFMZ7GAMHHPJAPANEFSGQKEJ4SQW4IYKE7FOA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.944012296 (61h2m9.666575337s) [noise=true , address=45.91.116.9:4263 , piece_id=NRLM5DFHZL5RZM2DUIYXKDLOIMIBIZKPSXAUG3FCDYTCA4CF54WA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.990566447 (61h2m9.713129477s) [noise=true , address=45.91.116.8:4045 , piece_id=TFKIFVA6FMBE63ILNB6QTJBHPLYL7PEKSTGZ3LWRFUST2RMNFPXA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.008214494 (61h2m9.730777544s) [noise=true , address=45.91.116.4:4939 , piece_id=YTQBXZKLMAB7KGB7BMYM7TA6J4NVMU6N7GUTAIK77DLRGY2D2A6A, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.053756088 (61h2m9.776319138s) [noise=true , address=45.91.116.11:4117 , piece_id=ZPVFASEDZOJ7PH2NOB3S4NIRH5TDW7OGTJUOLCQCZAAPGKUB35LA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.099507814 (61h2m9.822070884s) [noise=true , address=45.91.116.5:4299 , piece_id=N3JVJ3VWTNEEML7PPGERQ632YVEY6R5DVCPJ7YOPCKJO5EEQLLGQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:01.567939563 (61h2m11.290502604s) [noise=true , address=77.90.31.12:4590 , piece_id=ZCZF62HABMW4EVT4OBMWCPFAFMHY6LPYM7QJVFOBAH2HHIZT2UMA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:26.756358587 (61h2m36.478921627s) [noise=true , address=104.36.82.2:4732 , piece_id=O5UZK5W5HRROTFV45TPI35OLW2XLSVO4PPVCHQZVCNJYTMZ4YFDA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756404715 (61h2m36.478967755s) [noise=true , address=77.90.31.15:4698 , piece_id=Y4YNEMHV3VTRZUUZFSTZTHC2HIMFPYUTPYLRYZ6NCGOI34NQXPDQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756436546 (61h2m36.478999586s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXDQGDHP4NI633USSPA2NW4SBKW4OGUG764B5EXA6AKUULYLEAQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756470811 (61h2m36.479033851s) [noise=true , address=104.36.82.13:4186 , piece_id=FOWJ63ZEQLGVLC5UCVPHU2DNGY2POPR2XSH4VK6KORUWRFZNRHYA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756503444 (61h2m36.479066494s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:35416->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756520356 (61h2m36.479083386s) [seg_pos={0 0}, upload=58529]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:38:26.982864478 (61h2m36.705427518s) [upload=58529]: Done uploading segments... Jul 27 00:38:26.982888764 (61h2m36.705451804s) [upload=58529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.227817254s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX...] duration: 7.354361816s FAIL (2025-07-27 00:38:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.354385191s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXX] duration: 30.24140986s FAIL (2025-07-27 00:38:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241437193s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236805445s FAIL (2025-07-27 00:38:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== CPCXNZT353J6BFDAZRY7E3FMGZZFJA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371291336 (61h1m12.093854376s) [noise=true , address=79.127.205.235:4662 , piece_id=KVNXPF65ZPGYW4JGBJILVDIRBM3OKNBMWBWHKHZJO7N6AXA2CA2A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371592963 (61h1m12.094156003s) [noise=true , address=79.127.205.246:4258 , piece_id=LT7IFH6N2J4PY2OMU7PDXYOT25Q7P7X5QPJEURBPRPPBB7CXORZA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371694417 (61h1m12.094257457s) [noise=true , address=77.90.31.16:4766 , piece_id=5MICPG3KLZAM6DI6W7H2YMRESTWBEQCVGMEGFDV4B6SPGEIOVTFA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371834706 (61h1m12.094397746s) [noise=true , address=104.36.82.52:4102 , piece_id=47QXMWNPJLWVTXES2B4LEUNI3HTOJ4COJP32TOINJT5AU4DBKNRA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372017967 (61h1m12.094581007s) [noise=true , address=45.91.116.9:4277 , piece_id=ODGEJBFLQWQX6SDVJHZHEMZZD6C5BQHQGX7U3CUJV3P7XVFWDSEA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372110914 (61h1m12.094673964s) [noise=true , address=104.36.82.46:4502 , piece_id=LXCIOQEKNVPBYHR65OKHQMBRBFWFL7EPU4S3NOBS4AYWTLVLNOFA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372272053 (61h1m12.094835093s) [noise=true , address=77.90.31.13:4659 , piece_id=5UZGQZOUA3JNHDEKYEIVTDCFCZHL6ZXLEDQ42G72ZW37PFZVV6RQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372384097 (61h1m12.094947147s) [noise=true , address=79.127.219.38:4001 , piece_id=JRLPBPI3XKWDWGASGVC3TK7KZ5JK7VZNMKXBS7LHXXB4Z56GRUNQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372512643 (61h1m12.095075683s) [noise=true , address=45.91.116.11:4130 , piece_id=ND3PCYSBUZW7BRVBR3ENGIOAUKFUZ2G4U3PT35376UTYZLWBEVFQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372731301 (61h1m12.095294342s) [noise=true , address=156.146.43.225:4461 , piece_id=NAMVLJXO22XN3ALJOBQ3ZIA4RPZRQV2TAI5BU2YS7R75765GIK7A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373009924 (61h1m12.095572975s) [noise=true , address=104.36.82.6:4897 , piece_id=634ESXLDRPWNR76NDDV7CQ6RWGIA4KEECQ57JCA5SRHSDZBN24MQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373279280 (61h1m12.09584232s) [noise=true , address=104.36.82.55:4196 , piece_id=KANVAUP5O7ESWJSN7FY2RWFDZZV4JKFEXBUIGTZ766BWO6HNHFGA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373519629 (61h1m12.096082669s) [noise=true , address=104.36.82.42:4824 , piece_id=4PSTOFHG3OYPJYJWSWWUVYNDMBMD6IBM7QVAM2YAB2HMYOACNU7A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373763376 (61h1m12.096326416s) [noise=true , address=77.90.31.10:4535 , piece_id=6BUGLJUZBLBMNS7TQPZVNWWZ7HNBPNT4BMTBHGXIZ34AFPYNQGJQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374040696 (61h1m12.096603736s) [noise=true , address=104.36.82.47:4548 , piece_id=DZCJFKR5D4QJMHCQOPK4RPYFUIHB5Z2XPAW6ODZWANG5TCHLJ2OQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374283070 (61h1m12.09684612s) [noise=true , address=77.90.31.11:4579 , piece_id=L537CARZN4TF4SP4A7FDCOKZPR2ZQF63L77BT464FK2T3FZQQPIA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374522789 (61h1m12.097085829s) [noise=true , address=104.36.82.53:4132 , piece_id=LPW3JBPUYJD3VBMYTFFKH6PPKJG2GKCKWCW4D4VIILR5SLDXA35A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:03.391908288 (61h1m13.114471328s) [noise=true , address=156.146.43.225:4461 , piece_id=NAMVLJXO22XN3ALJOBQ3ZIA4RPZRQV2TAI5BU2YS7R75765GIK7A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.394174424 (61h1m13.116737464s) [noise=true , address=77.90.31.13:4659 , piece_id=5UZGQZOUA3JNHDEKYEIVTDCFCZHL6ZXLEDQ42G72ZW37PFZVV6RQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.445295333 (61h1m13.167858363s) [noise=true , address=77.90.31.10:4535 , piece_id=6BUGLJUZBLBMNS7TQPZVNWWZ7HNBPNT4BMTBHGXIZ34AFPYNQGJQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.487138281 (61h1m13.209701321s) [upload=58514]: Next returned nil segment Jul 27 00:37:03.487155714 (61h1m13.209718754s) [upload=58514]: Waiting for error group managing segments... Jul 27 00:37:03.803770128 (61h1m13.526333168s) [noise=true , address=79.127.205.235:4662 , piece_id=KVNXPF65ZPGYW4JGBJILVDIRBM3OKNBMWBWHKHZJO7N6AXA2CA2A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:03.805587385 (61h1m13.528150425s) [noise=true , address=79.127.205.246:4258 , piece_id=LT7IFH6N2J4PY2OMU7PDXYOT25Q7P7X5QPJEURBPRPPBB7CXORZA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.359774673 (61h1m14.082337714s) [noise=true , address=77.90.31.16:4766 , piece_id=5MICPG3KLZAM6DI6W7H2YMRESTWBEQCVGMEGFDV4B6SPGEIOVTFA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.448912750 (61h1m14.17147579s) [noise=true , address=79.127.219.38:4001 , piece_id=JRLPBPI3XKWDWGASGVC3TK7KZ5JK7VZNMKXBS7LHXXB4Z56GRUNQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.514013877 (61h1m14.236576917s) [noise=true , address=77.90.31.11:4579 , piece_id=L537CARZN4TF4SP4A7FDCOKZPR2ZQF63L77BT464FK2T3FZQQPIA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.722635389 (61h1m14.445198469s) [noise=true , address=45.91.116.9:4277 , piece_id=ODGEJBFLQWQX6SDVJHZHEMZZD6C5BQHQGX7U3CUJV3P7XVFWDSEA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:14.313642296 (61h1m24.036205326s) [noise=true , address=45.91.116.11:4130 , piece_id=ND3PCYSBUZW7BRVBR3ENGIOAUKFUZ2G4U3PT35376UTYZLWBEVFQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:24.029047161 (61h1m33.751610201s) [noise=true , address=77.90.31.12:4616 , piece_id=Z262EKINXVBQ44VXHM4NFOCPCXNZT353J6BFDAZRY7E3FMGZZFJA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:38:01.262280310 (61h2m10.98484337s) [noise=true , address=104.36.82.53:4132 , piece_id=LPW3JBPUYJD3VBMYTFFKH6PPKJG2GKCKWCW4D4VIILR5SLDXA35A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262498608 (61h2m10.985061658s) [noise=true , address=104.36.82.42:4824 , piece_id=4PSTOFHG3OYPJYJWSWWUVYNDMBMD6IBM7QVAM2YAB2HMYOACNU7A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262522544 (61h2m10.985085624s) [noise=true , address=104.36.82.47:4548 , piece_id=DZCJFKR5D4QJMHCQOPK4RPYFUIHB5Z2XPAW6ODZWANG5TCHLJ2OQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: write tcp 188.166.225.170:33332->104.36.82.47:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262574223 (61h2m10.985137263s) [noise=true , address=104.36.82.55:4196 , piece_id=KANVAUP5O7ESWJSN7FY2RWFDZZV4JKFEXBUIGTZ766BWO6HNHFGA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262620200 (61h2m10.98518324s) [noise=true , address=104.36.82.46:4502 , piece_id=LXCIOQEKNVPBYHR65OKHQMBRBFWFL7EPU4S3NOBS4AYWTLVLNOFA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 188.166.225.170:50744->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262635329 (61h2m10.985198379s) [noise=true , address=104.36.82.52:4102 , piece_id=47QXMWNPJLWVTXES2B4LEUNI3HTOJ4COJP32TOINJT5AU4DBKNRA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262712547 (61h2m10.985275607s) [noise=true , address=104.36.82.6:4897 , piece_id=634ESXLDRPWNR76NDDV7CQ6RWGIA4KEECQ57JCA5SRHSDZBN24MQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12qtpRnAcGiHmdR6J6ysdwaLPvejnCvpKm6MggqsezhbZ14Z5Bt, address:104.36.82.6:4897): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262732946 (61h2m10.985295986s) [seg_pos={0 0}, upload=58514]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:38:01.497391895 (61h2m11.219954935s) [upload=58514]: Done uploading segments... Jul 27 00:38:01.497424437 (61h2m11.219987487s) [upload=58514]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236819251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX..] duration: 22.129969487s FAIL (2025-07-27 00:37:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.12999262s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X] duration: 1m0.00122038s FAIL (2025-07-27 00:37:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001245308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................X................] duration: 1.858463773s FAIL (2025-07-27 00:37:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.85850476s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXX] duration: 30.23069085s FAIL (2025-07-27 00:37:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=5L6NBZQC3UFLXPX4EHH23C3SK3HAW5AAW2EXCYOKL56RUNKTM35A, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:55.282591689 (61h1m5.005154719s) [noise=true , address=104.36.82.12:4150 , piece_id=AFI5POE2LKVDPVWDI7OJK3346AUQZCSEFTG46V4UJJWV6BQDYO5Q, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op Jul 27 00:36:55.282616637 (61h1m5.005179677s) [seg_pos={0 0}, upload=58501]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:36:55.525510482 (61h1m5.248073522s) [upload=58501]: Done uploading segments... Jul 27 00:36:55.525557102 (61h1m5.248120142s) [upload=58501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:36:56.392498939 (61h1m6.115061979s) [upload=58512]: Starting upload Jul 27 00:36:56.392550648 (61h1m6.115113688s) [upload=58512]: Done starting upload Jul 27 00:36:56.392602407 (61h1m6.115165447s) [upload=58512]: Uploading segments... Jul 27 00:36:56.392727386 (61h1m6.115290436s) [upload=58512]: Got next segment. Inline: false Jul 27 00:36:56.392777331 (61h1m6.115340371s) [upload=58512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:36:56.392781760 (61h1m6.1153448s) [upload=58512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:36:56.627999299 (61h1m6.350562339s) [seg_pos={0 0}, upload=58512]: Begin upload segment... Jul 27 00:36:56.628042671 (61h1m6.350605711s) [seg_pos={0 0}, upload=58512]: Done begin upload segment. Jul 27 00:36:56.628066487 (61h1m6.350629517s) [noise=true , address=45.91.116.11:4117 , piece_id=JCULBSIB6PTBVHZGO7FKUYSFMLEQKELOONMAQOKLRCDURJGPEDXA, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628071366 (61h1m6.350634406s) [noise=true , address=45.91.116.14:4191 , piece_id=EAM6OKUB2X4476FI34CFKKLPDUZUD2YOEFCW6NP5CECXHLTRE6XQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628402299 (61h1m6.350965339s) [noise=true , address=45.91.116.9:4262 , piece_id=L5FSTJS2CDWEZNKFY5A2GOLKVPE62EE5VFFYOBHGOW72VBMWAVXQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628425574 (61h1m6.350988614s) [noise=true , address=77.90.31.10:4519 , piece_id=P7N7UYQTO5RBPXHALMTNFYOR74QZ6Q44IDJRAHQ6NVTP7UZVYMKQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628672517 (61h1m6.351235547s) [noise=true , address=45.91.116.8:4047 , piece_id=QCGIAUYY37ONH5SRCACOILUGRVVGLCGNUVIVLFAKOES4GK3FH2QA, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628717453 (61h1m6.351280493s) [noise=true , address=45.91.116.6:4973 , piece_id=ALDGNEUOI2HIIRXR22R7OEH5IPZ2NIGKC3UFDBEJYFME6JPJQTIQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628916373 (61h1m6.351479413s) [noise=true , address=45.91.116.12:4226 , piece_id=AVQHSRAPFYPCBEIUAYT4YDJG27HROZ4Y7SMDH44IYV3GULQCPBJQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629024931 (61h1m6.351587971s) [noise=true , address=45.91.116.9:4263 , piece_id=BPRWHLOQWUXBRQUQN74CH65LQYX3V2ZBRBDDX2J2ARVDJQQTZWQQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629166371 (61h1m6.351729401s) [noise=true , address=45.91.116.13:4154 , piece_id=6TKB3GMZ4GSYRWCPBZP4NAM5B5YKCLB3I4SYYT2T6W2MKGJRNCHQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629363038 (61h1m6.351926088s) [noise=true , address=104.36.82.6:4873 , piece_id=AT47NWH3OPO46WJBDPAX7SBVNQPEVQBDFESTVRYBDCBKS2RTXCUQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629446738 (61h1m6.352009778s) [noise=true , address=45.91.116.4:4937 , piece_id=CLDU4KN4RDF7MU4GFILV3K36OANM7I3MGS55JJVIIKFHH6H36DBQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629654796 (61h1m6.352217836s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629696646 (61h1m6.352259686s) [noise=true , address=104.36.82.15:4258 , piece_id=UCUPLNJLKE5ST7DZ2EBPQDA2TH5E7J7GG7INICENTEIXX22F6GRQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629897721 (61h1m6.352460751s) [noise=true , address=45.91.116.10:4083 , piece_id=LRZFDCUGV2YX5Q7GNBXXF7NVWRZ6WMRDBTM7VENE3WBPW6WIGRHQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629916967 (61h1m6.352480007s) [noise=true , address=77.90.31.21:4877 , piece_id=3DCP3VATQUXELXXCGQNHGC36Z45ZLFNWBFXBYAH4G6ZKNDKSVGGQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.630155995 (61h1m6.352719035s) [noise=true , address=45.91.116.4:4938 , piece_id=W775KX7CIUM4ALKYVFUWMTINTJ5TS76TECGTGPGMU2FKUJOMNGGQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:58.359101342 (61h1m8.081664382s) [upload=58512]: Next returned nil segment Jul 27 00:36:58.359120808 (61h1m8.081683848s) [upload=58512]: Waiting for error group managing segments... Jul 27 00:36:58.636622779 (61h1m8.359185819s) [noise=true , address=45.91.116.8:4047 , piece_id=QCGIAUYY37ONH5SRCACOILUGRVVGLCGNUVIVLFAKOES4GK3FH2QA, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.718032168 (61h1m8.440595208s) [noise=true , address=45.91.116.4:4937 , piece_id=CLDU4KN4RDF7MU4GFILV3K36OANM7I3MGS55JJVIIKFHH6H36DBQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.718474775 (61h1m8.441037815s) [noise=true , address=45.91.116.12:4226 , piece_id=AVQHSRAPFYPCBEIUAYT4YDJG27HROZ4Y7SMDH44IYV3GULQCPBJQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.722680021 (61h1m8.445243061s) [noise=true , address=45.91.116.10:4083 , piece_id=LRZFDCUGV2YX5Q7GNBXXF7NVWRZ6WMRDBTM7VENE3WBPW6WIGRHQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.736511374 (61h1m8.459074414s) [noise=true , address=45.91.116.9:4262 , piece_id=L5FSTJS2CDWEZNKFY5A2GOLKVPE62EE5VFFYOBHGOW72VBMWAVXQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.736719141 (61h1m8.459282181s) [noise=true , address=45.91.116.6:4973 , piece_id=ALDGNEUOI2HIIRXR22R7OEH5IPZ2NIGKC3UFDBEJYFME6JPJQTIQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.782897574 (61h1m8.505460614s) [noise=true , address=45.91.116.14:4191 , piece_id=EAM6OKUB2X4476FI34CFKKLPDUZUD2YOEFCW6NP5CECXHLTRE6XQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.790103231 (61h1m8.512666271s) [noise=true , address=45.91.116.13:4154 , piece_id=6TKB3GMZ4GSYRWCPBZP4NAM5B5YKCLB3I4SYYT2T6W2MKGJRNCHQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.791579856 (61h1m8.514142886s) [noise=true , address=45.91.116.4:4938 , piece_id=W775KX7CIUM4ALKYVFUWMTINTJ5TS76TECGTGPGMU2FKUJOMNGGQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.804398562 (61h1m8.526961592s) [noise=true , address=45.91.116.9:4263 , piece_id=BPRWHLOQWUXBRQUQN74CH65LQYX3V2ZBRBDDX2J2ARVDJQQTZWQQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.809874008 (61h1m8.532437048s) [noise=true , address=45.91.116.11:4117 , piece_id=JCULBSIB6PTBVHZGO7FKUYSFMLEQKELOONMAQOKLRCDURJGPEDXA, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:37:00.167671442 (61h1m9.890234482s) [noise=true , address=77.90.31.10:4519 , piece_id=P7N7UYQTO5RBPXHALMTNFYOR74QZ6Q44IDJRAHQ6NVTP7UZVYMKQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:37:25.753434958 (61h1m35.475997998s) [noise=true , address=104.36.82.6:4873 , piece_id=AT47NWH3OPO46WJBDPAX7SBVNQPEVQBDFESTVRYBDCBKS2RTXCUQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:25.753452572 (61h1m35.476015612s) [noise=true , address=104.36.82.15:4258 , piece_id=UCUPLNJLKE5ST7DZ2EBPQDA2TH5E7J7GG7INICENTEIXX22F6GRQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op Jul 27 00:37:25.753478221 (61h1m35.476041261s) [noise=true , address=77.90.31.21:4877 , piece_id=3DCP3VATQUXELXXCGQNHGC36Z45ZLFNWBFXBYAH4G6ZKNDKSVGGQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:25.753540791 (61h1m35.476103861s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; context deadline exceeded Jul 27 00:37:25.753560599 (61h1m35.476123629s) [seg_pos={0 0}, upload=58512]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:37:25.982334529 (61h1m35.704897569s) [upload=58512]: Done uploading segments... Jul 27 00:37:25.982363554 (61h1m35.704926594s) [upload=58512]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230708824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XX] duration: 30.305544949s FAIL (2025-07-27 00:37:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.305568213s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX.] duration: 6.241579084s FAIL (2025-07-27 00:37:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.241594212s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)