Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 6.646744358s FAIL (2025-08-10 22:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.64676658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X..X...............X.......X.X.....X.X........] duration: 7.925714829s FAIL (2025-08-10 22:48:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.925742782s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X...............................X......................] duration: 1.903017325s FAIL (2025-08-10 22:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.903049798s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 7.585026434s FAIL (2025-08-10 22:48:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.585047044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.324978246s FAIL (2025-08-10 22:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 8.325009136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X..X...............X.......X.X.....X.X.......] duration: 8.408562743s FAIL (2025-08-10 22:47:46 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 8.408591096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X...............................X.....................] duration: 1.98746558s FAIL (2025-08-10 22:47:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.987497281s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X..X...............X.......X.X.....X.X......] duration: 7.72027555s FAIL (2025-08-10 22:47:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.720300569s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 7.354018267s FAIL (2025-08-10 22:47:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.354044398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.036162611s FAIL (2025-08-10 22:47:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 8.036184874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X...............................X....................] duration: 1.906372124s FAIL (2025-08-10 22:47:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.906393196s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.605681087s FAIL (2025-08-10 22:46:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.60571346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 6.467148498s FAIL (2025-08-10 22:46:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.467216708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X...............................X...................] duration: 1.93967841s FAIL (2025-08-10 22:46:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.939724609s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X..X...............X.......X.X.....X.X.....] duration: 6.49937754s FAIL (2025-08-10 22:46:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 6.499474026s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 8.020851095s FAIL (2025-08-10 22:46:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.020881994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.304485244s FAIL (2025-08-10 22:45:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.304515762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X..X...............X.......X.X.....X.X....] duration: 6.39146199s FAIL (2025-08-10 22:45:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 6.391489453s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X...............................X..................] duration: 1.928279056s FAIL (2025-08-10 22:45:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.928307841s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 7.520323401s FAIL (2025-08-10 22:45:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.520350313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X...............................X.................] duration: 1.852996656s FAIL (2025-08-10 22:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.853042324s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X..X...............X.......X.X.....X.X...] duration: 6.726013994s FAIL (2025-08-10 22:44:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 6.726037621s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.585759974s FAIL (2025-08-10 22:44:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.585780434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X...............................X................] duration: 1.923122503s FAIL (2025-08-10 22:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.92315204s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 6.343496864s FAIL (2025-08-10 22:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.343523754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X..X...............X.......X.X.....X.X..] duration: 7.353090563s FAIL (2025-08-10 22:43:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.353116041s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.332054266s FAIL (2025-08-10 22:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.332095645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X...............................X...............] duration: 1.854926601s FAIL (2025-08-10 22:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.854953383s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X..X...............X.......X.X.....X.X.] duration: 8.762391349s FAIL (2025-08-10 22:43:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 8.762425476s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.513264552s FAIL (2025-08-10 22:43:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.513286494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 6.867350982s FAIL (2025-08-10 22:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.86738032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X...............................X..............] duration: 1.828377413s FAIL (2025-08-10 22:42:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.828400558s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 6.759172334s FAIL (2025-08-10 22:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.75919598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X..X...............X.......X.X.....X.X] duration: 4.650599216s FAIL (2025-08-10 22:42:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 4.650625757s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X......................X..X...............X.......X.X.....X.] duration: 22.497467892s FAIL (2025-08-10 22:42:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== Aug 10 22:40:27.126518057 (53h14m21.557598178s) [upload=55518]: Starting upload Aug 10 22:40:27.126569426 (53h14m21.557649557s) [upload=55518]: Done starting upload Aug 10 22:40:27.126676280 (53h14m21.557756411s) [upload=55518]: Uploading segments... Aug 10 22:40:27.126783976 (53h14m21.557864117s) [upload=55518]: Got next segment. Inline: false Aug 10 22:40:27.126830005 (53h14m21.557910136s) [upload=55518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:40:27.126834854 (53h14m21.557914985s) [upload=55518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:40:27.580617804 (53h14m22.011697935s) [seg_pos={0 0}, upload=55518]: Begin upload segment... Aug 10 22:40:27.580700092 (53h14m22.011780213s) [noise=true , address=104.36.82.5:4845 , piece_id=Y6N5HYZN6CD6CIUGAX4EASE6F3BRWWGBC77IFNZWX5WAV6GBDFAA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581261044 (53h14m22.012341175s) [noise=true , address=64.49.13.154:4929 , piece_id=5VWSLQIMG5PGXXDXVINQZN5H45NIPREVJ5DPNUDM5LXUVYJRY6ZA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581581919 (53h14m22.01266204s) [noise=true , address=104.36.82.16:4328 , piece_id=2Y3RYE7V6PCXYFFPG4Y5WDFNFQ7P27SDOHO4QTD7CZJYT4BZ274A, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581916148 (53h14m22.012996269s) [noise=true , address=104.36.82.44:4883 , piece_id=FFKZW3ICUC63EAG5VNA6FL3FEDJALEK3Y2NRIXBEJ7RYWSJRNIVA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581945715 (53h14m22.013025916s) [noise=true , address=104.36.82.17:4344 , piece_id=53UEHGXOYZPKMRIM4IAMRD4IYRVKVV46GIJCHVETDWMWP53XJWDA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.582828704 (53h14m22.013908835s) [noise=true , address=104.36.82.56:4242 , piece_id=EEARRK2H25I2OEO2PUK3XBJMKJPR6C5TA4TABSN7JFCCNYC2DOLQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.583525337 (53h14m22.014605468s) [noise=true , address=104.36.82.11:4118 , piece_id=SQL6VZCRVVPXM6FHDWFSZR2WR5TOIOLLPG6O3VK27EOTR6CW4LJQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.584126558 (53h14m22.015206689s) [noise=true , address=79.127.219.48:4075 , piece_id=EOSUV2ER7BZUUATVJMATGA2SM3X6BTQE7QZS6JNNBSQKVUNC6KWQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.584626685 (53h14m22.015706816s) [noise=true , address=104.36.82.60:4345 , piece_id=IZJH2TPMXP4GE2W4DQKJTRYJ2QR3PISU7U7KZTKPZIFXFAR3WSSA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.585164443 (53h14m22.016244574s) [noise=true , address=104.36.82.9:4009 , piece_id=ZWXQXNY6XHT6O5VG5HEOMZR7EAWMWWXPR43V6RWXPHQVMO32PPFQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.585598393 (53h14m22.016678524s) [noise=true , address=104.36.82.13:4197 , piece_id=YSHR4DFX7B4AEGKN5UTIB7VNAFW5QVATWBJJGPVZVAM57XZUQZEA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.586171720 (53h14m22.017251862s) [noise=true , address=104.36.82.35:4616 , piece_id=LEBIO7DMBLHUFKMOASSUJR7ZEBMVMIQ3VWMAY6IAQW24IYYHIBSQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.586594529 (53h14m22.01767465s) [noise=true , address=104.36.82.18:4371 , piece_id=U64GEW5D55AFAHJFTF5QXROV5MVZX4N7XEXTMNZVHPG2YCESFDJQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587007910 (53h14m22.018088041s) [noise=true , address=104.36.82.59:4316 , piece_id=IABK4RFLCQSE7K4MSBYNLKE6T7EZBKZCK2YTLCB5KR3UFTMPH7CQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587358020 (53h14m22.018438151s) [noise=true , address=64.49.13.26:4948 , piece_id=KTYFQBHXEAUOFXTPPIBOAEU6UMG5MDJ7HQKLBFUI4R2XV3GJRDTQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587709623 (53h14m22.018789764s) [noise=true , address=45.91.116.7:4023 , piece_id=LJRXKTP3AXZ4ZKZTECKRDDKAJAN7MP6JRWHWXG6NQQ5BFYDAHFAQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.588056056 (53h14m22.019136187s) [noise=true , address=104.36.82.37:4659 , piece_id=HCEXN5IXTMH6UFOLF5RFPKGOQWDLPSLTG6DSBOASGXJ3J43YGK3Q, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.588387741 (53h14m22.019467872s) [noise=true , address=104.36.82.53:4145 , piece_id=PHASULXZNNFKC2DLICXLZBWDSCM3ETJMHP4RBSK2PFIVE6G6AFCQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.589635367 (53h14m22.020715509s) [seg_pos={0 0}, upload=55518]: Done begin upload segment. Aug 10 22:40:28.743296351 (53h14m23.174376482s) [upload=55518]: Next returned nil segment Aug 10 22:40:28.743311509 (53h14m23.17439164s) [upload=55518]: Waiting for error group managing segments... Aug 10 22:40:29.054538702 (53h14m23.485618833s) [noise=true , address=64.49.13.154:4929 , piece_id=5VWSLQIMG5PGXXDXVINQZN5H45NIPREVJ5DPNUDM5LXUVYJRY6ZA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:29.055588952 (53h14m23.486669073s) [noise=true , address=64.49.13.26:4948 , piece_id=KTYFQBHXEAUOFXTPPIBOAEU6UMG5MDJ7HQKLBFUI4R2XV3GJRDTQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:29.526055023 (53h14m23.957135154s) [noise=true , address=79.127.219.48:4075 , piece_id=EOSUV2ER7BZUUATVJMATGA2SM3X6BTQE7QZS6JNNBSQKVUNC6KWQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:36.956062994 (53h14m31.387143125s) [noise=true , address=45.91.116.7:4023 , piece_id=LJRXKTP3AXZ4ZKZTECKRDDKAJAN7MP6JRWHWXG6NQQ5BFYDAHFAQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:41:26.484520977 (53h15m20.915601098s) [noise=true , address=104.36.82.35:4616 , piece_id=LEBIO7DMBLHUFKMOASSUJR7ZEBMVMIQ3VWMAY6IAQW24IYYHIBSQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1rDCZdLm3zP15D7MoanzkotJ4HJjvj1GS1R9aJdQmDJpvEboin, address:104.36.82.35:4616): protocol: expected piece hash; context deadline exceeded; EOF Aug 10 22:41:26.484865595 (53h15m20.915945717s) [noise=true , address=104.36.82.44:4883 , piece_id=FFKZW3ICUC63EAG5VNA6FL3FEDJALEK3Y2NRIXBEJ7RYWSJRNIVA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN, address:104.36.82.44:4883): protocol: rpc: dial tcp 104.36.82.44:4883: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.484947903 (53h15m20.916028034s) [noise=true , address=104.36.82.53:4145 , piece_id=PHASULXZNNFKC2DLICXLZBWDSCM3ETJMHP4RBSK2PFIVE6G6AFCQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: rpc: dial tcp 104.36.82.53:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.484998961 (53h15m20.916079092s) [noise=true , address=104.36.82.56:4242 , piece_id=EEARRK2H25I2OEO2PUK3XBJMKJPR6C5TA4TABSN7JFCCNYC2DOLQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485031232 (53h15m20.916111353s) [noise=true , address=104.36.82.11:4118 , piece_id=SQL6VZCRVVPXM6FHDWFSZR2WR5TOIOLLPG6O3VK27EOTR6CW4LJQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:104.36.82.11:4118): protocol: rpc: dial tcp 104.36.82.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485065919 (53h15m20.916146051s) [noise=true , address=104.36.82.9:4009 , piece_id=ZWXQXNY6XHT6O5VG5HEOMZR7EAWMWWXPR43V6RWXPHQVMO32PPFQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: write tcp 188.166.225.170:42956->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485095326 (53h15m20.916175457s) [noise=true , address=104.36.82.13:4197 , piece_id=YSHR4DFX7B4AEGKN5UTIB7VNAFW5QVATWBJJGPVZVAM57XZUQZEA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: rpc: dial tcp 104.36.82.13:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485121376 (53h15m20.916201507s) [noise=true , address=104.36.82.18:4371 , piece_id=U64GEW5D55AFAHJFTF5QXROV5MVZX4N7XEXTMNZVHPG2YCESFDJQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:104.36.82.18:4371): protocol: rpc: dial tcp 104.36.82.18:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485163656 (53h15m20.916243777s) [noise=true , address=104.36.82.59:4316 , piece_id=IABK4RFLCQSE7K4MSBYNLKE6T7EZBKZCK2YTLCB5KR3UFTMPH7CQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:124WiP4Umxs7oHKtCpaQrysLxRDsrmqkyTCs2Ps38ZjM2LPvVhn, address:104.36.82.59:4316): protocol: rpc: dial tcp 104.36.82.59:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485184676 (53h15m20.916264807s) [noise=true , address=104.36.82.17:4344 , piece_id=53UEHGXOYZPKMRIM4IAMRD4IYRVKVV46GIJCHVETDWMWP53XJWDA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12rKbSAu7Sp15uZTHHovG8xAwAW3t2wRMYYGuZgEboCFQTauskq, address:104.36.82.17:4344): protocol: rpc: dial tcp 104.36.82.17:4344: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485208813 (53h15m20.916288944s) [noise=true , address=104.36.82.5:4845 , piece_id=Y6N5HYZN6CD6CIUGAX4EASE6F3BRWWGBC77IFNZWX5WAV6GBDFAA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube, address:104.36.82.5:4845): protocol: write tcp 188.166.225.170:56994->104.36.82.5:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485232238 (53h15m20.916312369s) [noise=true , address=104.36.82.60:4345 , piece_id=IZJH2TPMXP4GE2W4DQKJTRYJ2QR3PISU7U7KZTKPZIFXFAR3WSSA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: rpc: dial tcp 104.36.82.60:4345: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485246534 (53h15m20.916326656s) [noise=true , address=104.36.82.16:4328 , piece_id=2Y3RYE7V6PCXYFFPG4Y5WDFNFQ7P27SDOHO4QTD7CZJYT4BZ274A, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:4328): protocol: rpc: dial tcp 104.36.82.16:4328: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485263177 (53h15m20.916343298s) [noise=true , address=104.36.82.37:4659 , piece_id=HCEXN5IXTMH6UFOLF5RFPKGOQWDLPSLTG6DSBOASGXJ3J43YGK3Q, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12UBuWLogtN8mv4nwXDpzGdQtbKDz1EfVENSjXC54QdFp2fJg99, address:104.36.82.37:4659): protocol: read tcp 188.166.225.170:52874->104.36.82.37:4659: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485293725 (53h15m20.916373856s) [seg_pos={0 0}, upload=55518]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Aug 10 22:41:26.720368853 (53h15m21.151449004s) [upload=55518]: Done uploading segments... Aug 10 22:41:26.720411684 (53h15m21.151491825s) [upload=55518]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 22.4974929s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 6.202299277s FAIL (2025-08-10 22:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.202314126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 7.289457215s FAIL (2025-08-10 22:41:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 7.289485719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.236248168s FAIL (2025-08-10 22:41:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.236269929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X...............................X.............] duration: 1.925202905s FAIL (2025-08-10 22:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.925229445s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X......................X..X...............X.......X.X.....X] duration: 1m0.237485526s FAIL (2025-08-10 22:41:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== Aug 10 22:40:27.126518057 (53h14m21.557598178s) [upload=55518]: Starting upload Aug 10 22:40:27.126569426 (53h14m21.557649557s) [upload=55518]: Done starting upload Aug 10 22:40:27.126676280 (53h14m21.557756411s) [upload=55518]: Uploading segments... Aug 10 22:40:27.126783976 (53h14m21.557864117s) [upload=55518]: Got next segment. Inline: false Aug 10 22:40:27.126830005 (53h14m21.557910136s) [upload=55518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:40:27.126834854 (53h14m21.557914985s) [upload=55518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:40:27.580617804 (53h14m22.011697935s) [seg_pos={0 0}, upload=55518]: Begin upload segment... Aug 10 22:40:27.580700092 (53h14m22.011780213s) [noise=true , address=104.36.82.5:4845 , piece_id=Y6N5HYZN6CD6CIUGAX4EASE6F3BRWWGBC77IFNZWX5WAV6GBDFAA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581261044 (53h14m22.012341175s) [noise=true , address=64.49.13.154:4929 , piece_id=5VWSLQIMG5PGXXDXVINQZN5H45NIPREVJ5DPNUDM5LXUVYJRY6ZA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581581919 (53h14m22.01266204s) [noise=true , address=104.36.82.16:4328 , piece_id=2Y3RYE7V6PCXYFFPG4Y5WDFNFQ7P27SDOHO4QTD7CZJYT4BZ274A, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581916148 (53h14m22.012996269s) [noise=true , address=104.36.82.44:4883 , piece_id=FFKZW3ICUC63EAG5VNA6FL3FEDJALEK3Y2NRIXBEJ7RYWSJRNIVA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.581945715 (53h14m22.013025916s) [noise=true , address=104.36.82.17:4344 , piece_id=53UEHGXOYZPKMRIM4IAMRD4IYRVKVV46GIJCHVETDWMWP53XJWDA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.582828704 (53h14m22.013908835s) [noise=true , address=104.36.82.56:4242 , piece_id=EEARRK2H25I2OEO2PUK3XBJMKJPR6C5TA4TABSN7JFCCNYC2DOLQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.583525337 (53h14m22.014605468s) [noise=true , address=104.36.82.11:4118 , piece_id=SQL6VZCRVVPXM6FHDWFSZR2WR5TOIOLLPG6O3VK27EOTR6CW4LJQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.584126558 (53h14m22.015206689s) [noise=true , address=79.127.219.48:4075 , piece_id=EOSUV2ER7BZUUATVJMATGA2SM3X6BTQE7QZS6JNNBSQKVUNC6KWQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.584626685 (53h14m22.015706816s) [noise=true , address=104.36.82.60:4345 , piece_id=IZJH2TPMXP4GE2W4DQKJTRYJ2QR3PISU7U7KZTKPZIFXFAR3WSSA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.585164443 (53h14m22.016244574s) [noise=true , address=104.36.82.9:4009 , piece_id=ZWXQXNY6XHT6O5VG5HEOMZR7EAWMWWXPR43V6RWXPHQVMO32PPFQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.585598393 (53h14m22.016678524s) [noise=true , address=104.36.82.13:4197 , piece_id=YSHR4DFX7B4AEGKN5UTIB7VNAFW5QVATWBJJGPVZVAM57XZUQZEA, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.586171720 (53h14m22.017251862s) [noise=true , address=104.36.82.35:4616 , piece_id=LEBIO7DMBLHUFKMOASSUJR7ZEBMVMIQ3VWMAY6IAQW24IYYHIBSQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.586594529 (53h14m22.01767465s) [noise=true , address=104.36.82.18:4371 , piece_id=U64GEW5D55AFAHJFTF5QXROV5MVZX4N7XEXTMNZVHPG2YCESFDJQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587007910 (53h14m22.018088041s) [noise=true , address=104.36.82.59:4316 , piece_id=IABK4RFLCQSE7K4MSBYNLKE6T7EZBKZCK2YTLCB5KR3UFTMPH7CQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587358020 (53h14m22.018438151s) [noise=true , address=64.49.13.26:4948 , piece_id=KTYFQBHXEAUOFXTPPIBOAEU6UMG5MDJ7HQKLBFUI4R2XV3GJRDTQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.587709623 (53h14m22.018789764s) [noise=true , address=45.91.116.7:4023 , piece_id=LJRXKTP3AXZ4ZKZTECKRDDKAJAN7MP6JRWHWXG6NQQ5BFYDAHFAQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.588056056 (53h14m22.019136187s) [noise=true , address=104.36.82.37:4659 , piece_id=HCEXN5IXTMH6UFOLF5RFPKGOQWDLPSLTG6DSBOASGXJ3J43YGK3Q, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.588387741 (53h14m22.019467872s) [noise=true , address=104.36.82.53:4145 , piece_id=PHASULXZNNFKC2DLICXLZBWDSCM3ETJMHP4RBSK2PFIVE6G6AFCQ, seg_pos={0 0}, upload=55518]: Uploading piece... Aug 10 22:40:27.589635367 (53h14m22.020715509s) [seg_pos={0 0}, upload=55518]: Done begin upload segment. Aug 10 22:40:28.743296351 (53h14m23.174376482s) [upload=55518]: Next returned nil segment Aug 10 22:40:28.743311509 (53h14m23.17439164s) [upload=55518]: Waiting for error group managing segments... Aug 10 22:40:29.054538702 (53h14m23.485618833s) [noise=true , address=64.49.13.154:4929 , piece_id=5VWSLQIMG5PGXXDXVINQZN5H45NIPREVJ5DPNUDM5LXUVYJRY6ZA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:29.055588952 (53h14m23.486669073s) [noise=true , address=64.49.13.26:4948 , piece_id=KTYFQBHXEAUOFXTPPIBOAEU6UMG5MDJ7HQKLBFUI4R2XV3GJRDTQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:29.526055023 (53h14m23.957135154s) [noise=true , address=79.127.219.48:4075 , piece_id=EOSUV2ER7BZUUATVJMATGA2SM3X6BTQE7QZS6JNNBSQKVUNC6KWQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:40:36.956062994 (53h14m31.387143125s) [noise=true , address=45.91.116.7:4023 , piece_id=LJRXKTP3AXZ4ZKZTECKRDDKAJAN7MP6JRWHWXG6NQQ5BFYDAHFAQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: <nil> Aug 10 22:41:26.484520977 (53h15m20.915601098s) [noise=true , address=104.36.82.35:4616 , piece_id=LEBIO7DMBLHUFKMOASSUJR7ZEBMVMIQ3VWMAY6IAQW24IYYHIBSQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1rDCZdLm3zP15D7MoanzkotJ4HJjvj1GS1R9aJdQmDJpvEboin, address:104.36.82.35:4616): protocol: expected piece hash; context deadline exceeded; EOF Aug 10 22:41:26.484865595 (53h15m20.915945717s) [noise=true , address=104.36.82.44:4883 , piece_id=FFKZW3ICUC63EAG5VNA6FL3FEDJALEK3Y2NRIXBEJ7RYWSJRNIVA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN, address:104.36.82.44:4883): protocol: rpc: dial tcp 104.36.82.44:4883: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.484947903 (53h15m20.916028034s) [noise=true , address=104.36.82.53:4145 , piece_id=PHASULXZNNFKC2DLICXLZBWDSCM3ETJMHP4RBSK2PFIVE6G6AFCQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: rpc: dial tcp 104.36.82.53:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.484998961 (53h15m20.916079092s) [noise=true , address=104.36.82.56:4242 , piece_id=EEARRK2H25I2OEO2PUK3XBJMKJPR6C5TA4TABSN7JFCCNYC2DOLQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485031232 (53h15m20.916111353s) [noise=true , address=104.36.82.11:4118 , piece_id=SQL6VZCRVVPXM6FHDWFSZR2WR5TOIOLLPG6O3VK27EOTR6CW4LJQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:104.36.82.11:4118): protocol: rpc: dial tcp 104.36.82.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485065919 (53h15m20.916146051s) [noise=true , address=104.36.82.9:4009 , piece_id=ZWXQXNY6XHT6O5VG5HEOMZR7EAWMWWXPR43V6RWXPHQVMO32PPFQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: write tcp 188.166.225.170:42956->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485095326 (53h15m20.916175457s) [noise=true , address=104.36.82.13:4197 , piece_id=YSHR4DFX7B4AEGKN5UTIB7VNAFW5QVATWBJJGPVZVAM57XZUQZEA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: rpc: dial tcp 104.36.82.13:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485121376 (53h15m20.916201507s) [noise=true , address=104.36.82.18:4371 , piece_id=U64GEW5D55AFAHJFTF5QXROV5MVZX4N7XEXTMNZVHPG2YCESFDJQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:104.36.82.18:4371): protocol: rpc: dial tcp 104.36.82.18:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485163656 (53h15m20.916243777s) [noise=true , address=104.36.82.59:4316 , piece_id=IABK4RFLCQSE7K4MSBYNLKE6T7EZBKZCK2YTLCB5KR3UFTMPH7CQ, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:124WiP4Umxs7oHKtCpaQrysLxRDsrmqkyTCs2Ps38ZjM2LPvVhn, address:104.36.82.59:4316): protocol: rpc: dial tcp 104.36.82.59:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485184676 (53h15m20.916264807s) [noise=true , address=104.36.82.17:4344 , piece_id=53UEHGXOYZPKMRIM4IAMRD4IYRVKVV46GIJCHVETDWMWP53XJWDA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12rKbSAu7Sp15uZTHHovG8xAwAW3t2wRMYYGuZgEboCFQTauskq, address:104.36.82.17:4344): protocol: rpc: dial tcp 104.36.82.17:4344: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485208813 (53h15m20.916288944s) [noise=true , address=104.36.82.5:4845 , piece_id=Y6N5HYZN6CD6CIUGAX4EASE6F3BRWWGBC77IFNZWX5WAV6GBDFAA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube, address:104.36.82.5:4845): protocol: write tcp 188.166.225.170:56994->104.36.82.5:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485232238 (53h15m20.916312369s) [noise=true , address=104.36.82.60:4345 , piece_id=IZJH2TPMXP4GE2W4DQKJTRYJ2QR3PISU7U7KZTKPZIFXFAR3WSSA, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: rpc: dial tcp 104.36.82.60:4345: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485246534 (53h15m20.916326656s) [noise=true , address=104.36.82.16:4328 , piece_id=2Y3RYE7V6PCXYFFPG4Y5WDFNFQ7P27SDOHO4QTD7CZJYT4BZ274A, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:4328): protocol: rpc: dial tcp 104.36.82.16:4328: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485263177 (53h15m20.916343298s) [noise=true , address=104.36.82.37:4659 , piece_id=HCEXN5IXTMH6UFOLF5RFPKGOQWDLPSLTG6DSBOASGXJ3J43YGK3Q, seg_pos={0 0}, upload=55518]: Done uploading piece. err: ecclient: upload failed (node:12UBuWLogtN8mv4nwXDpzGdQtbKDz1EfVENSjXC54QdFp2fJg99, address:104.36.82.37:4659): protocol: read tcp 188.166.225.170:52874->104.36.82.37:4659: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:26.485293725 (53h15m20.916373856s) [seg_pos={0 0}, upload=55518]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Aug 10 22:41:26.720368853 (53h15m21.151449004s) [upload=55518]: Done uploading segments... Aug 10 22:41:26.720411684 (53h15m21.151491825s) [upload=55518]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.237514291s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.776105893s FAIL (2025-08-10 22:41:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 5.776131852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............................X............] duration: 1.928054681s FAIL (2025-08-10 22:40:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.928087674s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 25.281484332s FAIL (2025-08-10 22:40:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 25.281511124s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.240453739s FAIL (2025-08-10 22:40:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 6.82.20:4451 , piece_id=72XL7Q5DXF36ILLXXE3HXK2LG3GPAFC4ZQIAIBY7DKXDDQBYDCUA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.495899834 (53h13m44.926979955s) [noise=true , address=104.36.82.3:4787 , piece_id=QUMSTI4C6SPGW4A7CDSBKAMU7EGN6CY4ZML5NJJBPJW4EZ3JGURQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.495911336 (53h13m44.926991457s) [noise=true , address=104.36.82.12:4183 , piece_id=ZYCM7IWCWDEC354HJP5TLYEFYBZCGUZB4N7YOW2RHN6BZFSVVAYQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.496216450 (53h13m44.927296571s) [noise=true , address=104.36.82.18:4381 , piece_id=NIBDX2MOIA6MZIYDPZK637YW6XMQGNGQ4YB3STGTTK7V7HEA5DOQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.496222131 (53h13m44.927302262s) [noise=true , address=104.36.82.2:4744 , piece_id=OYABWN3XXXULFZ2FF2KWQFXEJ7POSV75CDD5SJ42UDLH7ZKMDUHA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.496506765 (53h13m44.927586896s) [noise=true , address=45.91.116.5:4304 , piece_id=3IGEFJCYMKBARS33BOWAJKZUXZNOI55MW3I3HQCJWKELZSE6CP7Q, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.496647825 (53h13m44.927727956s) [noise=true , address=104.36.82.45:4912 , piece_id=WRP6SDF5X43DFZVV2DYOIRSYGVPPQSIFS4G6N5HFL35YOZRAZTJA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.497013706 (53h13m44.928093827s) [noise=true , address=104.36.82.61:4407 , piece_id=LVRKNMMMOSAHMMY5IFIWUTRRFL2QKIJQKPTIXKC6QFM5ATMWMSGQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.497255387 (53h13m44.928335508s) [noise=true , address=45.91.116.7:4022 , piece_id=6CMDUIGBR6UCNZJQINGJ322KJ64FB42HYSTWF2L7KCNSZJIMLPLQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.497537067 (53h13m44.928617208s) [noise=true , address=104.36.82.34:4565 , piece_id=B2CF2I3PJJIOKSJ7TE73GNTDA7O72NBH3ZLWB5OPOWXEYJQVGLOA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.497752369 (53h13m44.92883249s) [noise=true , address=193.149.164.143:4697 , piece_id=72DLEJ65Y5P3F4MCNVP3UMGZ5OPAQRIAQDRLF3X4OKNE4IVTOTUQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.498028878 (53h13m44.929109009s) [noise=true , address=104.36.82.58:4271 , piece_id=GDS7QJYKKEIYS6QROVJQ3OMTULX7LFLV73UQUJQEDM4YAZGBUNZA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.498440676 (53h13m44.929520807s) [noise=true , address=104.36.82.47:4546 , piece_id=6DAXVTAVZSVU43NJU4UD7OGRTAJWFLFP3XBTZMRKM7TRVDHEVZNA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.503399554 (53h13m44.934479685s) [noise=true , address=64.49.13.27:4957 , piece_id=T2EFDZIBRB7AA2ANKO66EXIQYDLDJUEGFBAXGDO74SHBNEXQC5JA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.507118429 (53h13m44.93819855s) [noise=true , address=104.36.82.39:4742 , piece_id=WKUVI67YICFBZLZSJ5NKK25HDCF4E674IP62KWLHQTBLJSS5OGMQ, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.512418469 (53h13m44.943498601s) [noise=true , address=104.36.82.54:4162 , piece_id=6OAPGDP72QSNZ5OTF6PV4MDYTKWQSXGJWWIUBNVI6MQ6DYCQ5RMA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:50.514617998 (53h13m44.945698129s) [noise=true , address=104.36.82.35:4582 , piece_id=S5EC6ARQYWK2RSYU6B7IPLQOVQ6JR4IFI2UH4JMYAG75F6OFYKUA, seg_pos={0 0}, upload=55508]: Uploading piece... Aug 10 22:39:51.627891386 (53h13m46.058971517s) [upload=55508]: Next returned nil segment Aug 10 22:39:51.627908510 (53h13m46.058988641s) [upload=55508]: Waiting for error group managing segments... Aug 10 22:39:51.933920132 (53h13m46.365000253s) [noise=true , address=64.49.13.27:4957 , piece_id=T2EFDZIBRB7AA2ANKO66EXIQYDLDJUEGFBAXGDO74SHBNEXQC5JA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:39:51.935797204 (53h13m46.366877326s) [noise=true , address=193.149.164.143:4697 , piece_id=72DLEJ65Y5P3F4MCNVP3UMGZ5OPAQRIAQDRLF3X4OKNE4IVTOTUQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:39:52.868204805 (53h13m47.299284937s) [noise=true , address=104.36.82.2:4744 , piece_id=OYABWN3XXXULFZ2FF2KWQFXEJ7POSV75CDD5SJ42UDLH7ZKMDUHA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:02.661918395 (53h13m57.092998536s) [noise=true , address=104.36.82.54:4162 , piece_id=6OAPGDP72QSNZ5OTF6PV4MDYTKWQSXGJWWIUBNVI6MQ6DYCQ5RMA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:21.141099143 (53h14m15.572179284s) [noise=true , address=104.36.82.34:4565 , piece_id=B2CF2I3PJJIOKSJ7TE73GNTDA7O72NBH3ZLWB5OPOWXEYJQVGLOA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:35.099045163 (53h14m29.530125294s) [noise=true , address=45.91.116.5:4304 , piece_id=3IGEFJCYMKBARS33BOWAJKZUXZNOI55MW3I3HQCJWKELZSE6CP7Q, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:35.628660997 (53h14m30.059741128s) [noise=true , address=104.36.82.47:4546 , piece_id=6DAXVTAVZSVU43NJU4UD7OGRTAJWFLFP3XBTZMRKM7TRVDHEVZNA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:36.187708804 (53h14m30.618788935s) [noise=true , address=104.36.82.39:4742 , piece_id=WKUVI67YICFBZLZSJ5NKK25HDCF4E674IP62KWLHQTBLJSS5OGMQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:45.802916298 (53h14m40.233996439s) [noise=true , address=45.91.116.7:4022 , piece_id=6CMDUIGBR6UCNZJQINGJ322KJ64FB42HYSTWF2L7KCNSZJIMLPLQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: <nil> Aug 10 22:40:49.410180203 (53h14m43.841260334s) [noise=true , address=104.36.82.44:4880 , piece_id=5EQSVTIZGQ4U3JCBWKQPXOQZJDBHBA6Z5C3NG7AGC2USPQXU5V4Q, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: read tcp 188.166.225.170:35842->104.36.82.44:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410226732 (53h14m43.841306863s) [noise=true , address=104.36.82.12:4183 , piece_id=ZYCM7IWCWDEC354HJP5TLYEFYBZCGUZB4N7YOW2RHN6BZFSVVAYQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410269463 (53h14m43.841349595s) [noise=true , address=104.36.82.61:4407 , piece_id=LVRKNMMMOSAHMMY5IFIWUTRRFL2QKIJQKPTIXKC6QFM5ATMWMSGQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:122k7b6wa2JhSXMVAhF4qPKA48cWkh3DZRyWuwPEt7Vp3ciofPF, address:104.36.82.61:4407): protocol: rpc: dial tcp 104.36.82.61:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410302818 (53h14m43.841382949s) [noise=true , address=104.36.82.20:4451 , piece_id=72XL7Q5DXF36ILLXXE3HXK2LG3GPAFC4ZQIAIBY7DKXDDQBYDCUA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:12fWecKRc63WWDEqfHomHdVKVaH3naVgZx6nhLZDJ4JnL3CfPQ, address:104.36.82.20:4451): protocol: rpc: dial tcp 104.36.82.20:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410320080 (53h14m43.841400202s) [noise=true , address=104.36.82.35:4582 , piece_id=S5EC6ARQYWK2RSYU6B7IPLQOVQ6JR4IFI2UH4JMYAG75F6OFYKUA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:104.36.82.35:4582): protocol: expected piece hash; context deadline exceeded; EOF Aug 10 22:40:49.410386658 (53h14m43.841466789s) [noise=true , address=104.36.82.18:4381 , piece_id=NIBDX2MOIA6MZIYDPZK637YW6XMQGNGQ4YB3STGTTK7V7HEA5DOQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:127RhN3jMBTt6LXsgym5RNMM4CvjhcSRhgjbRvSpE1TQTgQdJi9, address:104.36.82.18:4381): protocol: rpc: dial tcp 104.36.82.18:4381: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410422577 (53h14m43.841502698s) [noise=true , address=104.36.82.3:4787 , piece_id=QUMSTI4C6SPGW4A7CDSBKAMU7EGN6CY4ZML5NJJBPJW4EZ3JGURQ, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:104.36.82.3:4787): protocol: rpc: dial tcp 104.36.82.3:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410477352 (53h14m43.841557483s) [noise=true , address=104.36.82.58:4271 , piece_id=GDS7QJYKKEIYS6QROVJQ3OMTULX7LFLV73UQUJQEDM4YAZGBUNZA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: write tcp 188.166.225.170:37224->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410502470 (53h14m43.841582591s) [noise=true , address=104.36.82.45:4912 , piece_id=WRP6SDF5X43DFZVV2DYOIRSYGVPPQSIFS4G6N5HFL35YOZRAZTJA, seg_pos={0 0}, upload=55508]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:104.36.82.45:4912): protocol: read tcp 188.166.225.170:57244->104.36.82.45:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:49.410522989 (53h14m43.84160312s) [seg_pos={0 0}, upload=55508]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 10 22:40:49.648565184 (53h14m44.079645315s) [upload=55508]: Done uploading segments... Aug 10 22:40:49.648590352 (53h14m44.079670473s) [upload=55508]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.240497222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...............................X...........] duration: 1.865137486s FAIL (2025-08-10 22:40:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.865179928s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.241424886s FAIL (2025-08-10 22:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 30.241471744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............................X..........] duration: 1.836322889s FAIL (2025-08-10 22:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.836350022s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............................X.........] duration: 1.843860185s FAIL (2025-08-10 22:39:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.843883679s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X......................X..X...............X.......X.X.....] duration: 8.912893131s FAIL (2025-08-10 22:39:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:35:01.991180140 (53h8m56.422260271s) [upload=55416]: Starting upload Aug 10 22:35:01.991249994 (53h8m56.422330125s) [upload=55416]: Done starting upload Aug 10 22:35:01.991386756 (53h8m56.422466887s) [upload=55416]: Uploading segments... Aug 10 22:35:01.991539959 (53h8m56.42262009s) [upload=55416]: Got next segment. Inline: false Aug 10 22:35:01.991576168 (53h8m56.42265631s) [upload=55416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:35:01.991582009 (53h8m56.422662151s) [upload=55416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:35:02.207472010 (53h8m56.638552141s) [upload=55416]: Next segment err: metaclient: Too Many Requests Aug 10 22:35:02.207511495 (53h8m56.638591636s) [upload=55416]: Done uploading segments... Aug 10 22:35:02.207533417 (53h8m56.638613548s) [upload=55416]: Upload finished. err: metaclient: Too Many Requests Duration: 8.912920934s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............................X........] duration: 1.914773956s FAIL (2025-08-10 22:39:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.91480331s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)