Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX] duration: 30.227430826s FAIL (2025-07-27 00:04:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.227452569s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.......] duration: 10.157508953s FAIL (2025-07-27 00:04:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.157530234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234010391s FAIL (2025-07-27 00:04:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 429914s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296296305 (60h27m14.018859355s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296448477 (60h27m14.019011567s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296801893 (60h27m14.019364923s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296916422 (60h27m14.019479462s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297216586 (60h27m14.019779626s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297386952 (60h27m14.019949992s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297594509 (60h27m14.020157549s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297733676 (60h27m14.020296716s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297961742 (60h27m14.020524782s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298019222 (60h27m14.020582262s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298239854 (60h27m14.020802894s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298338854 (60h27m14.020901894s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298502326 (60h27m14.021065366s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298706026 (60h27m14.021269066s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298868206 (60h27m14.021431246s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.299167178 (60h27m14.021730229s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.299589586 (60h27m14.022152626s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:05.385782620 (60h27m15.10834566s) [upload=57985]: Next returned nil segment Jul 27 00:03:05.385828919 (60h27m15.108391969s) [upload=57985]: Waiting for error group managing segments... Jul 27 00:03:05.701031808 (60h27m15.423594848s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.701960976 (60h27m15.424524006s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.722094938 (60h27m15.444657968s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.743552713 (60h27m15.466115783s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:06.278512358 (60h27m16.001075398s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:08.340018352 (60h27m18.062581392s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:11.565721846 (60h27m21.288284886s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:13.988632217 (60h27m23.711195257s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:18.683164753 (60h27m28.405727793s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:30.503118221 (60h27m40.225681261s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:04:03.190575544 (60h28m12.913138584s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190656027 (60h28m12.913219067s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190692197 (60h28m12.913255237s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190745179 (60h28m12.913308219s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:37536->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190772331 (60h28m12.913335371s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190827717 (60h28m12.913390757s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190887290 (60h28m12.91345033s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190940382 (60h28m12.913503432s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190980088 (60h28m12.913543158s) [seg_pos={0 0}, upload=57985]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:04:03.422740258 (60h28m13.145303289s) [upload=57985]: Done uploading segments... Jul 27 00:04:03.422770667 (60h28m13.145333717s) [upload=57985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234036131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X......] duration: 32.74714383s FAIL (2025-07-27 00:03:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.747160172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......] duration: 6.248021118s FAIL (2025-07-27 00:03:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.248043411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.] duration: 5.808010915s FAIL (2025-07-27 00:03:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments... Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments... Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment... Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment. Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments... Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments... Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.808034329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.842571085s FAIL (2025-07-27 00:03:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.842596432s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 30.237202557s FAIL (2025-07-27 00:03:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments... Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments... Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment... Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment. Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments... Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments... Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.237224419s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.....] duration: 50.911700599s FAIL (2025-07-27 00:03:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.911720456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XX] duration: 30.236606689s FAIL (2025-07-27 00:03:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.236625776s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.011069868s FAIL (2025-07-27 00:03:04 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.011098783s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.261749221s FAIL (2025-07-27 00:02:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== [noise=true , address=104.36.82.52:4090 , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141114954 (60h26m2.863677994s) [noise=true , address=79.127.220.101:4048 , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141201730 (60h26m2.86376477s) [noise=true , address=104.36.82.47:4527 , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141378839 (60h26m2.863941869s) [noise=true , address=79.127.201.215:4356 , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141466387 (60h26m2.864029427s) [noise=true , address=104.36.82.46:4507 , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141624139 (60h26m2.864187179s) [noise=true , address=64.49.13.138:4482 , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141724240 (60h26m2.86428728s) [noise=true , address=79.127.205.245:4223 , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.141888344 (60h26m2.864451384s) [noise=true , address=104.36.82.54:4176 , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142007212 (60h26m2.864570252s) [noise=true , address=79.127.219.47:4048 , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142133743 (60h26m2.864696783s) [noise=true , address=77.90.31.15:4714 , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142227984 (60h26m2.864791024s) [noise=true , address=104.36.82.56:4244 , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142377239 (60h26m2.864940279s) [noise=true , address=104.36.82.7:4931 , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142443786 (60h26m2.865006826s) [noise=true , address=104.36.82.55:4216 , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142668477 (60h26m2.865231517s) [noise=true , address=45.91.116.10:4093 , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142690899 (60h26m2.865253929s) [noise=true , address=79.127.205.233:4639 , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:53.142868359 (60h26m2.865431399s) [noise=true , address=79.127.205.246:4286 , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Uploading piece... Jul 27 00:01:54.229399462 (60h26m3.951962502s) [upload=57968]: Next returned nil segment Jul 27 00:01:54.229417798 (60h26m3.951980828s) [upload=57968]: Waiting for error group managing segments... Jul 27 00:01:54.536628044 (60h26m4.259191084s) [noise=true , address=79.127.205.245:4223 , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:54.549230276 (60h26m4.271793316s) [noise=true , address=79.127.205.233:4639 , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:54.582728527 (60h26m4.305291567s) [noise=true , address=79.127.205.235:4685 , piece_id=3WBPT3C3LCIK2XY25Y5M7FO2CVM4MHZ4DUGBDVSS64EXDI766M3Q, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:54.609006844 (60h26m4.331569884s) [noise=true , address=79.127.205.246:4286 , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:55.637242753 (60h26m5.359805783s) [noise=true , address=64.49.13.138:4482 , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 27 00:01:55.706028969 (60h26m5.428592009s) [noise=true , address=45.91.116.10:4093 , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:57.495722163 (60h26m7.218285203s) [noise=true , address=79.127.219.47:4048 , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:58.749704002 (60h26m8.472267042s) [noise=true , address=79.127.220.101:4048 , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:01:58.762131119 (60h26m8.484694189s) [noise=true , address=79.127.201.215:4356 , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil> Jul 27 00:02:52.042028487 (60h27m1.764591517s) [noise=true , address=104.36.82.46:4507 , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042073464 (60h27m1.764636494s) [noise=true , address=104.36.82.39:4730 , piece_id=FDJH44ZA4Z6LXLNLQFLR5Y2B365AKY5J3X4G3M44SBI5PRMZI5FQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:104.36.82.39:4730): protocol: rpc: dial tcp 104.36.82.39:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042109052 (60h27m1.764672092s) [noise=true , address=104.36.82.52:4090 , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 188.166.225.170:32790->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042141123 (60h27m1.764704163s) [noise=true , address=104.36.82.54:4176 , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042285480 (60h27m1.76484852s) [noise=true , address=104.36.82.7:4931 , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL, address:104.36.82.7:4931): protocol: rpc: dial tcp 104.36.82.7:4931: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042300318 (60h27m1.764863358s) [noise=true , address=104.36.82.47:4527 , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042359180 (60h27m1.76492222s) [noise=true , address=104.36.82.55:4216 , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042365232 (60h27m1.764928272s) [noise=true , address=77.90.31.15:4714 , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: read tcp 188.166.225.170:58426->77.90.31.15:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042394899 (60h27m1.764957939s) [noise=true , address=104.36.82.56:4244 , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 188.166.225.170:53112->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:52.042423023 (60h27m1.764986063s) [seg_pos={0 0}, upload=57968]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:02:52.301418572 (60h27m2.023981632s) [upload=57968]: Done uploading segments... Jul 27 00:02:52.301447227 (60h27m2.024010267s) [upload=57968]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.261765342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.X] duration: 30.234335876s FAIL (2025-07-27 00:02:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.234357208s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXX] duration: 30.234483068s FAIL (2025-07-27 00:02:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 26260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:09.926623655 (60h26m19.649186695s) [upload=57970]: Starting upload Jul 27 00:02:09.926674673 (60h26m19.649237863s) [upload=57970]: Done starting upload Jul 27 00:02:09.926868904 (60h26m19.649431934s) [upload=57970]: Uploading segments... Jul 27 00:02:09.926902659 (60h26m19.649465689s) [upload=57970]: Got next segment. Inline: false Jul 27 00:02:09.926938417 (60h26m19.649501457s) [upload=57970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:09.926942434 (60h26m19.649505474s) [upload=57970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:10.159693471 (60h26m19.882256511s) [seg_pos={0 0}, upload=57970]: Begin upload segment... Jul 27 00:02:10.159735501 (60h26m19.882298541s) [seg_pos={0 0}, upload=57970]: Done begin upload segment. Jul 27 00:02:10.159757503 (60h26m19.882320543s) [noise=true , address=45.91.116.7:4010 , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.159819051 (60h26m19.882382091s) [noise=true , address=77.90.31.21:4877 , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.160178649 (60h26m19.882741679s) [noise=true , address=77.90.31.12:4590 , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.160383892 (60h26m19.882946922s) [noise=true , address=77.90.31.15:4699 , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.160457392 (60h26m19.883020432s) [noise=true , address=104.36.82.12:4149 , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.160718953 (60h26m19.883281993s) [noise=true , address=45.91.116.14:4189 , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.160747237 (60h26m19.883310277s) [noise=true , address=77.90.31.13:4627 , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161021091 (60h26m19.883584131s) [noise=true , address=104.36.82.14:4221 , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161057069 (60h26m19.883620109s) [noise=true , address=104.36.82.3:4765 , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161365109 (60h26m19.883928139s) [noise=true , address=45.91.116.13:4154 , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161371051 (60h26m19.883934091s) [noise=true , address=77.90.31.18:4805 , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161619686 (60h26m19.884182887s) [noise=true , address=104.36.82.16:4293 , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161651557 (60h26m19.884214597s) [noise=true , address=104.36.82.8:4969 , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161875946 (60h26m19.884438976s) [noise=true , address=104.36.82.7:4909 , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.161921764 (60h26m19.884484794s) [noise=true , address=45.91.116.8:4048 , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:10.162120615 (60h26m19.884683645s) [noise=true , address=45.91.116.10:4083 , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Uploading piece... Jul 27 00:02:11.908387111 (60h26m21.630950151s) [upload=57970]: Next returned nil segment Jul 27 00:02:11.908404243 (60h26m21.630967283s) [upload=57970]: Waiting for error group managing segments... Jul 27 00:02:12.184893462 (60h26m21.907456502s) [noise=true , address=45.91.116.7:4010 , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.185162627 (60h26m21.907725657s) [noise=true , address=77.90.31.18:4805 , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.230461587 (60h26m21.953024617s) [noise=true , address=45.91.116.10:4083 , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.256849564 (60h26m21.979412604s) [noise=true , address=45.91.116.13:4154 , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.262586270 (60h26m21.98514931s) [noise=true , address=77.90.31.13:4627 , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.283764439 (60h26m22.006327479s) [noise=true , address=77.90.31.21:4877 , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.353431912 (60h26m22.075994953s) [noise=true , address=45.91.116.8:4048 , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:12.482312036 (60h26m22.204875076s) [noise=true , address=45.91.116.14:4189 , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:13.616478367 (60h26m23.339041407s) [noise=true , address=77.90.31.12:4590 , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil> Jul 27 00:02:39.279759003 (60h26m49.002322084s) [noise=true , address=104.36.82.16:4293 , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.279833485 (60h26m49.002396525s) [noise=true , address=104.36.82.3:4765 , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:38402->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.279870246 (60h26m49.002433286s) [noise=true , address=104.36.82.14:4221 , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:44646->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.279916685 (60h26m49.002479725s) [noise=true , address=104.36.82.7:4909 , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.279955159 (60h26m49.002518199s) [noise=true , address=77.90.31.15:4699 , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.279987791 (60h26m49.002550821s) [noise=true , address=104.36.82.8:4969 , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41792->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280021024 (60h26m49.002584074s) [noise=true , address=104.36.82.12:4149 , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:33538->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments... Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.234525259s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X.....] duration: 6.329550432s FAIL (2025-07-27 00:02:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.329570659s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X....] duration: 32.728698643s FAIL (2025-07-27 00:02:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.728720753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.] duration: 7.023299469s FAIL (2025-07-27 00:02:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.023323927s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXX] duration: 30.232953879s FAIL (2025-07-27 00:02:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== n, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments... Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:01:35.539401218 (60h25m45.261964258s) [upload=57962]: Starting upload Jul 27 00:01:35.539463527 (60h25m45.262026557s) [upload=57962]: Done starting upload Jul 27 00:01:35.539640455 (60h25m45.262203496s) [upload=57962]: Uploading segments... Jul 27 00:01:35.539671956 (60h25m45.262234996s) [upload=57962]: Got next segment. Inline: false Jul 27 00:01:35.539692415 (60h25m45.262255445s) [upload=57962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:01:35.539696553 (60h25m45.262259593s) [upload=57962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:01:35.773032700 (60h25m45.49559574s) [seg_pos={0 0}, upload=57962]: Begin upload segment... Jul 27 00:01:35.773073748 (60h25m45.495636778s) [seg_pos={0 0}, upload=57962]: Done begin upload segment. Jul 27 00:01:35.773095660 (60h25m45.49565869s) [noise=true , address=77.90.31.21:4878 , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.773470618 (60h25m45.496033658s) [noise=true , address=45.91.116.7:4011 , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.773974211 (60h25m45.496537251s) [noise=true , address=77.90.31.19:4842 , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.774008596 (60h25m45.496571676s) [noise=true , address=104.36.82.11:4114 , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.774410496 (60h25m45.496973536s) [noise=true , address=77.90.31.16:4733 , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.774513844 (60h25m45.497076884s) [noise=true , address=45.91.116.13:4153 , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.774727582 (60h25m45.497290612s) [noise=true , address=45.91.116.13:4154 , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.774883190 (60h25m45.49744623s) [noise=true , address=45.91.116.4:4937 , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.775210867 (60h25m45.497773907s) [noise=true , address=45.91.116.14:4191 , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.775268106 (60h25m45.497831146s) [noise=true , address=104.36.82.12:4150 , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.775591756 (60h25m45.498154796s) [noise=true , address=45.91.116.6:4974 , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.775923420 (60h25m45.49848646s) [noise=true , address=45.91.116.9:4263 , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.776273600 (60h25m45.49883664s) [noise=true , address=77.90.31.15:4699 , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.776579666 (60h25m45.499142706s) [noise=true , address=77.90.31.10:4518 , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.776864541 (60h25m45.499427581s) [noise=true , address=77.90.31.12:4592 , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:35.777213028 (60h25m45.499776068s) [noise=true , address=104.36.82.3:4765 , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Uploading piece... Jul 27 00:01:37.516632604 (60h25m47.239195644s) [upload=57962]: Next returned nil segment Jul 27 00:01:37.516644707 (60h25m47.239207747s) [upload=57962]: Waiting for error group managing segments... Jul 27 00:01:37.793047892 (60h25m47.515610932s) [noise=true , address=45.91.116.13:4154 , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.877611314 (60h25m47.600174354s) [noise=true , address=45.91.116.14:4191 , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.886850622 (60h25m47.609413662s) [noise=true , address=45.91.116.7:4011 , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.887828463 (60h25m47.610391503s) [noise=true , address=45.91.116.6:4974 , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.894226173 (60h25m47.616789213s) [noise=true , address=77.90.31.21:4878 , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.954015056 (60h25m47.676578096s) [noise=true , address=45.91.116.4:4937 , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.954674427 (60h25m47.677237468s) [noise=true , address=45.91.116.13:4153 , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:37.968328110 (60h25m47.690891161s) [noise=true , address=45.91.116.9:4263 , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:39.090232961 (60h25m48.812796001s) [noise=true , address=77.90.31.12:4592 , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:39.245905354 (60h25m48.968468394s) [noise=true , address=77.90.31.15:4699 , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:01:39.909345256 (60h25m49.631908297s) [noise=true , address=77.90.31.19:4842 , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil> Jul 27 00:02:04.895836786 (60h26m14.618399826s) [noise=true , address=104.36.82.3:4765 , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:57196->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:04.895965743 (60h26m14.618528783s) [noise=true , address=77.90.31.10:4518 , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:04.895998005 (60h26m14.618561045s) [noise=true , address=104.36.82.11:4114 , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:37496->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:04.896051637 (60h26m14.618614678s) [noise=true , address=77.90.31.16:4733 , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45436->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:04.896131230 (60h26m14.61869428s) [noise=true , address=104.36.82.12:4150 , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:04.896156248 (60h26m14.618719288s) [seg_pos={0 0}, upload=57962]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:05.126235353 (60h26m14.848798393s) [upload=57962]: Done uploading segments... Jul 27 00:02:05.126260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.232968676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X] duration: 30.235725345s FAIL (2025-07-27 00:01:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235739561s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232792664s FAIL (2025-07-27 00:01:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , upload=57952]: Uploading piece... Jul 27 00:00:52.690310647 (60h25m2.412873687s) [noise=true , address=45.91.116.14:4194 , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.690629588 (60h25m2.413192628s) [noise=true , address=79.127.219.47:4048 , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.690755619 (60h25m2.413318659s) [noise=true , address=104.36.82.47:4526 , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.690902470 (60h25m2.41346551s) [noise=true , address=79.127.219.38:4018 , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691053178 (60h25m2.413616218s) [noise=true , address=104.36.82.52:4095 , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691357762 (60h25m2.413920802s) [noise=true , address=45.91.116.14:4206 , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691444227 (60h25m2.414007267s) [noise=true , address=104.36.82.53:4149 , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691650000 (60h25m2.41421304s) [noise=true , address=79.127.205.234:4608 , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691722409 (60h25m2.414285449s) [noise=true , address=79.127.220.97:4361 , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.691963541 (60h25m2.414526571s) [noise=true , address=77.90.31.10:4540 , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692069282 (60h25m2.414632322s) [noise=true , address=79.127.201.215:4352 , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692265959 (60h25m2.414828999s) [noise=true , address=64.49.13.138:4482 , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692443869 (60h25m2.415006899s) [noise=true , address=156.146.43.225:4448 , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692516518 (60h25m2.415079559s) [noise=true , address=64.49.13.138:4483 , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692758370 (60h25m2.41532141s) [noise=true , address=104.36.82.12:4168 , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692778279 (60h25m2.415341309s) [noise=true , address=104.36.82.46:4499 , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:52.692999933 (60h25m2.415562973s) [noise=true , address=104.36.82.49:4998 , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Uploading piece... Jul 27 00:00:53.707783843 (60h25m3.430346883s) [noise=true , address=79.127.219.47:4048 , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:53.808335285 (60h25m3.530898325s) [upload=57952]: Next returned nil segment Jul 27 00:00:53.808353410 (60h25m3.53091645s) [upload=57952]: Waiting for error group managing segments... Jul 27 00:00:53.894991431 (60h25m3.617554461s) [noise=true , address=79.127.220.97:4361 , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:54.294741272 (60h25m4.017304302s) [noise=true , address=79.127.205.234:4608 , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:54.372553368 (60h25m4.095116398s) [noise=true , address=64.49.13.138:4482 , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:54.551991192 (60h25m4.274554232s) [noise=true , address=79.127.219.38:4018 , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:54.786651822 (60h25m4.509214872s) [noise=true , address=45.91.116.14:4194 , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:56.389526811 (60h25m6.112089851s) [noise=true , address=45.91.116.14:4206 , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:57.469684549 (60h25m7.192247589s) [noise=true , address=156.146.43.225:4448 , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:57.904513196 (60h25m7.627076236s) [noise=true , address=79.127.201.214:4321 , piece_id=FQBU6OOVFDLBJCM44XHHWTGH3BOHMTC2XOBJH3L6DNOQAAFIQHOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:00:59.255459436 (60h25m8.978022476s) [noise=true , address=64.49.13.138:4483 , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:01:20.907151736 (60h25m30.629714776s) [noise=true , address=79.127.201.215:4352 , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil> Jul 27 00:01:51.585590532 (60h26m1.308153573s) [noise=true , address=104.36.82.53:4149 , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.585652411 (60h26m1.308215451s) [noise=true , address=77.90.31.10:4540 , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 188.166.225.170:37456->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.585885096 (60h26m1.308448136s) [noise=true , address=104.36.82.46:4499 , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:35230->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.585951222 (60h26m1.308514263s) [noise=true , address=104.36.82.52:4095 , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.585973626 (60h26m1.308536686s) [noise=true , address=104.36.82.12:4168 , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.586015997 (60h26m1.308579037s) [noise=true , address=104.36.82.49:4998 , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:104.36.82.49:4998): protocol: rpc: dial tcp 104.36.82.49:4998: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.586060892 (60h26m1.308623932s) [noise=true , address=104.36.82.47:4526 , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:33012->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:51.586097172 (60h26m1.308660243s) [seg_pos={0 0}, upload=57952]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:01:51.816141159 (60h26m1.538704189s) [upload=57952]: Done uploading segments... Jul 27 00:01:51.816172810 (60h26m1.53873585s) [upload=57952]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.232819476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.862142079s FAIL (2025-07-27 00:01:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.862169621s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X....] duration: 7.628504267s FAIL (2025-07-27 00:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.628519625s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...] duration: 22.886072806s FAIL (2025-07-27 00:01:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.886084397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX] duration: 30.23914945s FAIL (2025-07-27 00:01:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ent: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments... Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:01:05.077717119 (60h25m14.800280159s) [upload=57953]: Starting upload Jul 27 00:01:05.077788786 (60h25m14.800351827s) [upload=57953]: Done starting upload Jul 27 00:01:05.077967438 (60h25m14.800530478s) [upload=57953]: Uploading segments... Jul 27 00:01:05.078083480 (60h25m14.80064652s) [upload=57953]: Got next segment. Inline: false Jul 27 00:01:05.078125841 (60h25m14.800688881s) [upload=57953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:01:05.078131832 (60h25m14.800694872s) [upload=57953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:01:05.310276219 (60h25m15.03283926s) [seg_pos={0 0}, upload=57953]: Begin upload segment... Jul 27 00:01:05.310332166 (60h25m15.032895207s) [seg_pos={0 0}, upload=57953]: Done begin upload segment. Jul 27 00:01:05.310344830 (60h25m15.03290787s) [noise=true , address=77.90.31.12:4590 , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.310361623 (60h25m15.032924663s) [noise=true , address=104.36.82.10:4078 , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.310472134 (60h25m15.033035174s) [noise=true , address=45.91.116.11:4118 , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.310905342 (60h25m15.033468382s) [noise=true , address=104.36.82.16:4293 , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.311221878 (60h25m15.033784918s) [noise=true , address=104.36.82.8:4969 , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.311550277 (60h25m15.034113327s) [noise=true , address=45.91.116.9:4261 , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.311893673 (60h25m15.034456713s) [noise=true , address=45.91.116.10:4081 , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.312251939 (60h25m15.034814989s) [noise=true , address=77.90.31.15:4699 , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.312583363 (60h25m15.035146403s) [noise=true , address=45.91.116.6:4975 , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.312902724 (60h25m15.035465774s) [noise=true , address=104.36.82.2:4729 , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.313247313 (60h25m15.035810353s) [noise=true , address=45.91.116.8:4045 , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.313579468 (60h25m15.036142508s) [noise=true , address=45.91.116.13:4154 , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.314389629 (60h25m15.036952679s) [noise=true , address=104.36.82.7:4910 , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.315145515 (60h25m15.037708565s) [noise=true , address=77.90.31.19:4841 , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.316289474 (60h25m15.038852524s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:05.316765935 (60h25m15.039328985s) [noise=true , address=45.91.116.12:4227 , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Uploading piece... Jul 27 00:01:07.052152676 (60h25m16.774715716s) [upload=57953]: Next returned nil segment Jul 27 00:01:07.052172283 (60h25m16.774735323s) [upload=57953]: Waiting for error group managing segments... Jul 27 00:01:07.346109275 (60h25m17.068672315s) [noise=true , address=45.91.116.9:4261 , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.346622458 (60h25m17.069185498s) [noise=true , address=45.91.116.8:4045 , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.420735295 (60h25m17.143298325s) [noise=true , address=45.91.116.10:4081 , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.439976829 (60h25m17.162539869s) [noise=true , address=45.91.116.12:4227 , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.445103078 (60h25m17.167666118s) [noise=true , address=45.91.116.13:4154 , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.455448041 (60h25m17.178011081s) [noise=true , address=77.90.31.12:4590 , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.480860661 (60h25m17.203423732s) [noise=true , address=45.91.116.11:4118 , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.494123216 (60h25m17.216686246s) [noise=true , address=45.91.116.6:4975 , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:07.505061495 (60h25m17.227624535s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:08.692327849 (60h25m18.414890889s) [noise=true , address=77.90.31.19:4841 , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:08.870774407 (60h25m18.593337457s) [noise=true , address=77.90.31.15:4699 , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil> Jul 27 00:01:34.428248528 (60h25m44.150811578s) [noise=true , address=104.36.82.10:4078 , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:38342->104.36.82.10:4078: use of closed network connection; conn closed: op Jul 27 00:01:34.428315156 (60h25m44.150878226s) [noise=true , address=104.36.82.2:4729 , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:49100->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:34.428374339 (60h25m44.150937379s) [noise=true , address=104.36.82.8:4969 , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:01:34.428446246 (60h25m44.151009296s) [noise=true , address=104.36.82.16:4293 , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:34.428586766 (60h25m44.151149836s) [noise=true , address=104.36.82.7:4910 , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments... Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.239168005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.949046689s FAIL (2025-07-27 00:01:21 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.949078792s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..] duration: 29.413031147s FAIL (2025-07-27 00:01:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.413046005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXX] duration: 30.235163086s FAIL (2025-07-27 00:01:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments... Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:00:34.625856171 (60h24m44.348419211s) [upload=57946]: Starting upload Jul 27 00:00:34.625958327 (60h24m44.348521407s) [upload=57946]: Done starting upload Jul 27 00:00:34.626040784 (60h24m44.348603824s) [upload=57946]: Uploading segments... Jul 27 00:00:34.626264823 (60h24m44.348827863s) [upload=57946]: Got next segment. Inline: false Jul 27 00:00:34.626311623 (60h24m44.348874663s) [upload=57946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:00:34.626317774 (60h24m44.348880844s) [upload=57946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:00:34.865594065 (60h24m44.588157106s) [seg_pos={0 0}, upload=57946]: Begin upload segment... Jul 27 00:00:34.865647087 (60h24m44.588210127s) [seg_pos={0 0}, upload=57946]: Done begin upload segment. Jul 27 00:00:34.865670111 (60h24m44.588233141s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.865708626 (60h24m44.588271666s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866075106 (60h24m44.588638146s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866256885 (60h24m44.588819925s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866404937 (60h24m44.588967978s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866648443 (60h24m44.589211484s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866667950 (60h24m44.58923099s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866930773 (60h24m44.589493813s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867221260 (60h24m44.58978429s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867547112 (60h24m44.590110152s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867777634 (60h24m44.590340674s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868010669 (60h24m44.590573699s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868350270 (60h24m44.59091331s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868577584 (60h24m44.591140624s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868844886 (60h24m44.591407926s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.869181019 (60h24m44.591744049s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:36.571647152 (60h24m46.294210192s) [upload=57946]: Next returned nil segment Jul 27 00:00:36.571683281 (60h24m46.294246331s) [upload=57946]: Waiting for error group managing segments... Jul 27 00:00:36.869992438 (60h24m46.592555478s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.887108205 (60h24m46.609671245s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.907036524 (60h24m46.629599574s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.958513032 (60h24m46.681076072s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.973207827 (60h24m46.695770867s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.974133909 (60h24m46.696696949s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.977135692 (60h24m46.699698732s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.978503389 (60h24m46.701066429s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.016312929 (60h24m46.738875969s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.021475726 (60h24m46.744038766s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.036167487 (60h24m46.758730527s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:01:03.968881072 (60h25m13.691444112s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968922361 (60h25m13.691485401s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:53216->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968928994 (60h25m13.691492034s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:37916->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968952910 (60h25m13.69151595s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968971746 (60h25m13.691534786s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments... Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.235193273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X...] duration: 12.246817958s FAIL (2025-07-27 00:01:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.246844077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.99403249s FAIL (2025-07-27 00:00:50 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.99405258s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.] duration: 50.489387343s FAIL (2025-07-27 00:00:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.489408945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.] duration: 5.889514613s FAIL (2025-07-27 00:00:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.889541043s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236620041s FAIL (2025-07-27 00:00:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Jul 26 23:59:42.179886117 (60h23m51.902449157s) [seg_pos={0 0}, upload=57933]: Done begin upload segment. Jul 26 23:59:42.179910323 (60h23m51.902473363s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.179916154 (60h23m51.902479184s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.179962733 (60h23m51.902525763s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180351267 (60h23m51.902914327s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180808632 (60h23m51.903371672s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180835734 (60h23m51.903398774s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181092325 (60h23m51.903655355s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181114728 (60h23m51.903677758s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181342373 (60h23m51.903905403s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181360328 (60h23m51.903923368s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181588113 (60h23m51.904151153s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181851127 (60h23m51.904414167s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182136052 (60h23m51.904699082s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182437568 (60h23m51.905000598s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182650676 (60h23m51.905213716s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182935721 (60h23m51.905498761s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.183193584 (60h23m51.905756614s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.183391925 (60h23m51.905954965s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:43.285348328 (60h23m53.007911368s) [upload=57933]: Next returned nil segment Jul 26 23:59:43.285391911 (60h23m53.007954951s) [upload=57933]: Waiting for error group managing segments... Jul 26 23:59:43.598226841 (60h23m53.320789891s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.601115758 (60h23m53.323678798s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.642269155 (60h23m53.364832195s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.668410962 (60h23m53.390974012s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.708835476 (60h23m53.431398516s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.718426806 (60h23m53.440989846s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.097208810 (60h23m53.81977184s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.100794370 (60h23m53.823357411s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.132912180 (60h23m53.85547522s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.491805251 (60h23m54.214368291s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 27 00:00:18.017461556 (60h24m27.740024606s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 27 00:00:41.095307490 (60h24m50.81787053s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095364931 (60h24m50.817927971s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095404997 (60h24m50.817968037s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:40312->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095434554 (60h24m50.817997594s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095483527 (60h24m50.818046578s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:4483): protocol: rpc: dial tcp 104.36.82.21:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095546067 (60h24m50.818109107s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1yyUzkigTS9m3bDrZtAFL7tciNxpiCNswBSfBqXn1UW83GMLwi, address:104.36.82.10:4091): protocol: write tcp 188.166.225.170:53172->104.36.82.10:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095578399 (60h24m50.818141439s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: read tcp 188.166.225.170:48104->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095609017 (60h24m50.818172057s) [seg_pos={0 0}, upload=57933]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:00:41.330560987 (60h24m51.053124027s) [upload=57933]: Done uploading segments... Jul 27 00:00:41.330588720 (60h24m51.05315176s) [upload=57933]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.236633226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.949580404s FAIL (2025-07-27 00:00:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.949610942s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXX] duration: 30.2382672s FAIL (2025-07-27 00:00:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== [seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments... Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:59:53.265389883 (60h24m2.987952923s) [upload=57935]: Starting upload Jul 26 23:59:53.265451531 (60h24m2.988014571s) [upload=57935]: Done starting upload Jul 26 23:59:53.265547215 (60h24m2.988110255s) [upload=57935]: Uploading segments... Jul 26 23:59:53.265665881 (60h24m2.988228921s) [upload=57935]: Got next segment. Inline: false Jul 26 23:59:53.265695618 (60h24m2.988258648s) [upload=57935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:53.265700678 (60h24m2.988263718s) [upload=57935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:53.500569526 (60h24m3.223132566s) [seg_pos={0 0}, upload=57935]: Begin upload segment... Jul 26 23:59:53.500614643 (60h24m3.223177673s) [seg_pos={0 0}, upload=57935]: Done begin upload segment. Jul 26 23:59:53.500637407 (60h24m3.223200447s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.500687723 (60h24m3.223250763s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.500991074 (60h24m3.223554114s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501094200 (60h24m3.22365724s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501236714 (60h24m3.223799744s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501365379 (60h24m3.223928419s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501482303 (60h24m3.224045343s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501645385 (60h24m3.224208425s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501744114 (60h24m3.224307154s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501899962 (60h24m3.224463003s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502000194 (60h24m3.224563234s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502145102 (60h24m3.224708142s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502305698 (60h24m3.224868738s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502386763 (60h24m3.224949793s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502540347 (60h24m3.225103377s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502817999 (60h24m3.225381039s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:55.245077228 (60h24m4.967640258s) [upload=57935]: Next returned nil segment Jul 26 23:59:55.245105442 (60h24m4.967668482s) [upload=57935]: Waiting for error group managing segments... Jul 26 23:59:55.521486724 (60h24m5.244049764s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.535621258 (60h24m5.258184288s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.590385597 (60h24m5.312948627s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.596108576 (60h24m5.318671616s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.596498993 (60h24m5.319062023s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.607949671 (60h24m5.330512711s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.614678556 (60h24m5.337241596s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.632027378 (60h24m5.354590419s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.675652446 (60h24m5.398215486s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.807225580 (60h24m5.529788621s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 27 00:00:22.622487945 (60h24m32.345050985s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection Jul 27 00:00:22.622567417 (60h24m32.345130467s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622596592 (60h24m32.345159633s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622627972 (60h24m32.345191012s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622653822 (60h24m32.345216862s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622694220 (60h24m32.34525726s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments... Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.238292498s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X..] duration: 5.917415052s FAIL (2025-07-27 00:00:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.917451352s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX] duration: 30.244538304s FAIL (2025-07-27 00:00:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.244561219s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X] duration: 1m0.00115051s FAIL (2025-07-26 23:59:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001161703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXX] duration: 30.239655162s FAIL (2025-07-26 23:59:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 6 23:59:18.755564131 (60h23m28.478127171s) [upload=57925]: Starting upload Jul 26 23:59:18.755617964 (60h23m28.478181004s) [upload=57925]: Done starting upload Jul 26 23:59:18.755706744 (60h23m28.478269775s) [upload=57925]: Uploading segments... Jul 26 23:59:18.755814431 (60h23m28.478377471s) [upload=57925]: Got next segment. Inline: false Jul 26 23:59:18.755857272 (60h23m28.478420312s) [upload=57925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:18.755863173 (60h23m28.478426223s) [upload=57925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:18.989710817 (60h23m28.712273857s) [seg_pos={0 0}, upload=57925]: Begin upload segment... Jul 26 23:59:18.989748790 (60h23m28.71231183s) [seg_pos={0 0}, upload=57925]: Done begin upload segment. Jul 26 23:59:18.989766403 (60h23m28.712329443s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.989850995 (60h23m28.712414035s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990095784 (60h23m28.712658824s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990392171 (60h23m28.712955211s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990510397 (60h23m28.713073437s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990643101 (60h23m28.713206131s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990975377 (60h23m28.713538417s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991103472 (60h23m28.713666542s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991392835 (60h23m28.713955875s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991486916 (60h23m28.714049956s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991739719 (60h23m28.714302749s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991790737 (60h23m28.714353777s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991946946 (60h23m28.714509986s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992186614 (60h23m28.714749644s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992254975 (60h23m28.714818015s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992672093 (60h23m28.715235133s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:20.729860921 (60h23m30.452423961s) [upload=57925]: Next returned nil segment Jul 26 23:59:20.729875970 (60h23m30.45243901s) [upload=57925]: Waiting for error group managing segments... Jul 26 23:59:21.008316574 (60h23m30.730879614s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.008638079 (60h23m30.731201119s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.024163012 (60h23m30.746726052s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.160487698 (60h23m30.883050738s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.162286621 (60h23m30.884849661s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.461775184 (60h23m31.184338224s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:22.102625777 (60h23m31.825188817s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:22.398677264 (60h23m32.121240304s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:48.106089545 (60h23m57.828652585s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106139840 (60h23m57.82870288s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106225273 (60h23m57.828788313s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:35526->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106281591 (60h23m57.828844631s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106335895 (60h23m57.828898925s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106366715 (60h23m57.828929755s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:42844->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106414085 (60h23m57.828977115s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106446386 (60h23m57.829009426s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106468108 (60h23m57.829031148s) [seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments... Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.239673657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233995905s FAIL (2025-07-26 23:59:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.727576947 (60h22m51.450139977s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728137028 (60h22m51.450700068s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728432825 (60h22m51.450995865s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728479454 (60h22m51.451042494s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728761574 (60h22m51.451324614s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728956907 (60h22m51.451519947s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729045346 (60h22m51.451608386s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729528752 (60h22m51.452091802s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729779631 (60h22m51.452342661s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730008299 (60h22m51.452571339s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730341586 (60h22m51.452904626s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730668923 (60h22m51.453231963s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.731030434 (60h22m51.453593474s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.731448755 (60h22m51.454011785s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.732112174 (60h22m51.454675214s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.732720969 (60h22m51.455284019s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.733091287 (60h22m51.455654327s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:42.838834658 (60h22m52.561397688s) [upload=57914]: Next returned nil segment Jul 26 23:58:42.838851310 (60h22m52.56141435s) [upload=57914]: Waiting for error group managing segments... Jul 26 23:58:43.129009261 (60h22m52.851572311s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.138456927 (60h22m52.861019957s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.163842056 (60h22m52.886405096s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.583641918 (60h22m53.306204958s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.724061759 (60h22m53.446624799s) [noise=true , address=79.127.219.38:4010 , piece_id=XE3WNW4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.801955510 (60h22m53.52451855s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:44.316271165 (60h22m54.038834246s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:45.515232647 (60h22m55.237795707s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:08.772393964 (60h23m18.494957004s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:23.999032802 (60h23m33.721595842s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:40.631535807 (60h23m50.354098847s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op Jul 26 23:59:40.631613436 (60h23m50.354176476s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631773583 (60h23m50.354336623s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: write tcp 188.166.225.170:58646->104.36.82.53:4144: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631808780 (60h23m50.35437182s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: read tcp 188.166.225.170:44574->77.90.31.10:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631841593 (60h23m50.354404633s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:40786->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631884745 (60h23m50.354447785s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: rpc: dial tcp 104.36.82.56:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631949940 (60h23m50.35451298s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 188.166.225.170:59158->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631997791 (60h23m50.354560821s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.632018481 (60h23m50.354581521s) [seg_pos={0 0}, upload=57914]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:59:40.863499046 (60h23m50.586062086s) [upload=57914]: Done uploading segments... Jul 26 23:59:40.863526769 (60h23m50.586089799s) [upload=57914]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234016886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X.] duration: 5.91084676s FAIL (2025-07-26 23:59:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.91087298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.855632713s FAIL (2025-07-26 23:59:36 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.855661519s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration: 30.2216459s FAIL (2025-07-26 23:59:12 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 30.221660339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXX] duration: 30.232021331s FAIL (2025-07-26 23:59:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 1]: Done begin upload segment. Jul 26 23:58:37.367348346 (60h22m47.089911386s) [noise=true , address=104.36.82.15:4258 , piece_id=KJ4L6EGODVNLLI4Y5DOZLAGERZWCESEL32PRIDZAM2ZOHVKYBQTA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367354979 (60h22m47.089918019s) [noise=true , address=104.36.82.5:4839 , piece_id=VLZIP4UTTVBNFM5ELQBBMEPUWOOV5YRIZYDSELWQMGH7TTZYBPBQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367651737 (60h22m47.090214767s) [noise=true , address=77.90.31.16:4735 , piece_id=7YZHQWTT55WX7MFHC7RBZAPLK4V62DLHL5JMEYO5EEETJ4C4FEVQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367814779 (60h22m47.090377819s) [noise=true , address=77.90.31.18:4805 , piece_id=URT7OEWA2FBA2IXTSNJMGRJBICUYS6EC22RZAM5K26XBSQN62XEQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367941231 (60h22m47.090504271s) [noise=true , address=77.90.31.15:4698 , piece_id=JZJHPXR2TWAMUNEW6UL3UACSLBQRU3W4ELRGRA75Y43NUG5ZVFHQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368222088 (60h22m47.090785128s) [noise=true , address=77.90.31.21:4878 , piece_id=635AR4P2ZZRAS7MJ5ZKH4AIJSWFDY25CW3AKXEKLLU7XHFZTLZ4Q, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368453270 (60h22m47.09101631s) [noise=true , address=45.91.116.5:4299 , piece_id=VGWYIVCJW3U77BD6W2AELSY5DHHY56UHFLDHH23QNWKUQEY7WBTQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368674463 (60h22m47.091237503s) [noise=true , address=77.90.31.21:4879 , piece_id=IKN6NHWPFKD7VSYI76436BLJRMG7DSEHBQMLEBNKAEP5YA7LJOOA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368899384 (60h22m47.091462414s) [noise=true , address=45.91.116.6:4974 , piece_id=G7XXUR3ZAF2TMELMYKKLZGSRY26FJLZM5C6IIYUDFZ2UBUEWZ4SQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.369267328 (60h22m47.091830358s) [noise=true , address=104.36.82.3:4765 , piece_id=DGSSUPGJZGRLFKFRLGVKXXHUQ4KU3ANOYIKRRJAFWXGECUWL52AQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.369724183 (60h22m47.092287213s) [noise=true , address=104.36.82.12:4149 , piece_id=JMMSTQ36PQV7EYK7Y3XL3ES34VGOJP7UOFGVZKJHHY7XAORWGEAA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370241081 (60h22m47.092804121s) [noise=true , address=45.91.116.4:4937 , piece_id=5HO4BGY7L5T5MXZA2QHP6IAC5U3UQQLI2XXAOBITTX64XT5RJEEQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370447636 (60h22m47.093010676s) [noise=true , address=45.91.116.13:4153 , piece_id=MYPRMK4SWEZJHJVRIVWE257GCEFB6GXT2KBKSDRZUKX73QADOMYQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370637029 (60h22m47.093200069s) [noise=true , address=104.36.82.9:4007 , piece_id=TH2JYY4B3SOKN6T3WJUPHK25OYRDIEMIY3YMYMXHYVOFQNDUK4TQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370825760 (60h22m47.0933888s) [noise=true , address=104.36.82.11:4115 , piece_id=2HYDVHINWBEMOB7SG3WGEQVRSNXKLA4KWKB6JF65ZFMF7VBMGAPA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.371029900 (60h22m47.09359294s) [noise=true , address=77.90.31.16:4734 , piece_id=CICRVUAX6JG2H7GRCTFSHRUB3SD3LQZ3XC2N6PTB4THKRP2CB3GA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:59.233070760 (60h23m8.95563381s) [upload=57911]: Next returned nil segment Jul 26 23:58:59.233101228 (60h23m8.955664288s) [upload=57911]: Waiting for error group managing segments... Jul 26 23:58:59.569938991 (60h23m9.292502031s) [noise=true , address=45.91.116.5:4299 , piece_id=VGWYIVCJW3U77BD6W2AELSY5DHHY56UHFLDHH23QNWKUQEY7WBTQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:00.216394257 (60h23m9.938957297s) [noise=true , address=45.91.116.4:4937 , piece_id=5HO4BGY7L5T5MXZA2QHP6IAC5U3UQQLI2XXAOBITTX64XT5RJEEQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:00.738752451 (60h23m10.461315491s) [noise=true , address=45.91.116.6:4974 , piece_id=G7XXUR3ZAF2TMELMYKKLZGSRY26FJLZM5C6IIYUDFZ2UBUEWZ4SQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:02.413157868 (60h23m12.135720908s) [noise=true , address=77.90.31.15:4698 , piece_id=JZJHPXR2TWAMUNEW6UL3UACSLBQRU3W4ELRGRA75Y43NUG5ZVFHQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:03.173280099 (60h23m12.895843139s) [noise=true , address=45.91.116.13:4153 , piece_id=MYPRMK4SWEZJHJVRIVWE257GCEFB6GXT2KBKSDRZUKX73QADOMYQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:05.916467888 (60h23m15.639030928s) [noise=true , address=77.90.31.18:4805 , piece_id=URT7OEWA2FBA2IXTSNJMGRJBICUYS6EC22RZAM5K26XBSQN62XEQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:06.490595672 (60h23m16.213158702s) [noise=true , address=77.90.31.16:4735 , piece_id=7YZHQWTT55WX7MFHC7RBZAPLK4V62DLHL5JMEYO5EEETJ4C4FEVQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection Jul 26 23:59:06.490638293 (60h23m16.213201333s) [noise=true , address=77.90.31.21:4878 , piece_id=635AR4P2ZZRAS7MJ5ZKH4AIJSWFDY25CW3AKXEKLLU7XHFZTLZ4Q, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490670275 (60h23m16.213233305s) [noise=true , address=104.36.82.3:4765 , piece_id=DGSSUPGJZGRLFKFRLGVKXXHUQ4KU3ANOYIKRRJAFWXGECUWL52AQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490716222 (60h23m16.213279262s) [noise=true , address=104.36.82.11:4115 , piece_id=2HYDVHINWBEMOB7SG3WGEQVRSNXKLA4KWKB6JF65ZFMF7VBMGAPA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490745899 (60h23m16.213308939s) [noise=true , address=104.36.82.15:4258 , piece_id=KJ4L6EGODVNLLI4Y5DOZLAGERZWCESEL32PRIDZAM2ZOHVKYBQTA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490787809 (60h23m16.213350849s) [noise=true , address=104.36.82.12:4149 , piece_id=JMMSTQ36PQV7EYK7Y3XL3ES34VGOJP7UOFGVZKJHHY7XAORWGEAA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53152->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490795424 (60h23m16.213358464s) [noise=true , address=104.36.82.5:4839 , piece_id=VLZIP4UTTVBNFM5ELQBBMEPUWOOV5YRIZYDSELWQMGH7TTZYBPBQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490832855 (60h23m16.213395895s) [noise=true , address=77.90.31.21:4879 , piece_id=IKN6NHWPFKD7VSYI76436BLJRMG7DSEHBQMLEBNKAEP5YA7LJOOA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490841562 (60h23m16.213404602s) [noise=true , address=104.36.82.9:4007 , piece_id=TH2JYY4B3SOKN6T3WJUPHK25OYRDIEMIY3YMYMXHYVOFQNDUK4TQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490848946 (60h23m16.213411986s) [noise=true , address=77.90.31.16:4734 , piece_id=CICRVUAX6JG2H7GRCTFSHRUB3SD3LQZ3XC2N6PTB4THKRP2CB3GA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection Jul 26 23:59:06.490862603 (60h23m16.213425643s) [seg_pos={0 0}, upload=57911]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:59:06.720485622 (60h23m16.443048662s) [upload=57911]: Done uploading segments... Jul 26 23:59:06.720515039 (60h23m16.443078079s) [upload=57911]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.232044888s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X] duration: 30.000483269s FAIL (2025-07-26 23:59:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000506663s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX...] duration: 52.323428275s FAIL (2025-07-26 23:58:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.323448635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.854377712s FAIL (2025-07-26 23:58:45 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.854401578s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23001768s FAIL (2025-07-26 23:58:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 219.38:4006 , piece_id=H76UQJJ47D6F6AUDYYDVUJX75PLRWT5PUL75YI2W7MR7KXGGSBLA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.045959812 (60h21m47.768522852s) [noise=true , address=77.90.31.13:4629 , piece_id=ETIN7EO7P7GQ6O2RVKU5PRQAAKL47GCYD6AFEGUPI4VXILLPA6XA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046315172 (60h21m47.768878212s) [noise=true , address=79.127.220.98:4421 , piece_id=6IJ3UDSZ3OONAIM4HJOH7FZMWZJDNL3JIZW2FQVF7G3GCJO5372A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046427216 (60h21m47.768990256s) [noise=true , address=104.36.82.47:4519 , piece_id=M6PHGB34J74SAE2ZYI44NJ3V4HNMFWS75VTDTH3FVP7R6RJCGKVA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046666314 (60h21m47.769229344s) [noise=true , address=156.146.43.225:4448 , piece_id=SRHUMHAXD3JSLXF4QLOSSNWO2DO757HDGHQ3K73HG6XDLPVTIEYA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046714566 (60h21m47.769277596s) [noise=true , address=64.49.13.138:4483 , piece_id=2OXH6RJVAIYXZKRO2DC7AMEAEYK4WINBV52HAQUK7WBPXYKXKZRA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046939245 (60h21m47.769502286s) [noise=true , address=104.36.82.55:4197 , piece_id=WNB5MAAIUHWHWBBY5EJE56WHRN2HT4EZLOLCVLWORCMYHPVOQ26A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047020601 (60h21m47.769583641s) [noise=true , address=77.90.31.11:4585 , piece_id=56X44FTMER6FK647XF73E5EFU6NZQASTZH4YSK2Z77MQOJ635CUQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047209683 (60h21m47.769772713s) [noise=true , address=104.36.82.44:4894 , piece_id=YHXB52SOW4OPCIYBA5PQDGIBH3KLW3E2KWXGHXGHPHIP4WI7XKEA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047435766 (60h21m47.769998806s) [noise=true , address=79.127.205.246:4253 , piece_id=NATM45ZYE6H4RHXA4KNXW6CK7KQLPPRD4A7A6KOCJJWCSGYNJEEQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047656819 (60h21m47.770219859s) [noise=true , address=79.127.205.245:4240 , piece_id=IFFO3URWPDROY7WJE3ZNRHLRU67NNJAD2LLQHOHYCRLFAYMOK6YQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047876009 (60h21m47.770439049s) [noise=true , address=104.36.82.53:4153 , piece_id=WLBTOA5UHXE2WFXGJTJ2AYUZXWXY6ZYQ7MN4ZR44B4XRJ7PXV47A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.048093444 (60h21m47.770656484s) [noise=true , address=77.90.31.10:4544 , piece_id=TIZSESFHKJXQXE2P2ATUSPYJHFP5VD7FM6D4YF6Z3UEUGEALGAQQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.048329967 (60h21m47.770893007s) [noise=true , address=104.36.82.56:4262 , piece_id=ULQXGOENYL6GJKQHHQUXZMXHZBZGBPLE7NGM7CF64ZELWJF3QGMA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049040106 (60h21m47.771603136s) [noise=true , address=45.91.116.14:4194 , piece_id=DD2FUOO3O6YAYJZ23TYGROGF3BEJDCOX7Z6ZFTLR32V4DY23IY3Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049242322 (60h21m47.771805352s) [noise=true , address=77.90.31.16:4736 , piece_id=I2OIN67ZONMXFUFS4LFG6WJGH5S4E7VHW5DRANKCFCRJJ4EJZB2Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049432847 (60h21m47.771995887s) [noise=true , address=104.36.82.46:4501 , piece_id=YESAAOIXCYHVCWOZXKWO4NAIEMJGB5TA4A3GDPCB333C6KOVYR3Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049618332 (60h21m47.772181362s) [noise=true , address=104.36.82.54:4174 , piece_id=2HDRVUUXQZ7MQCZQNLO2FXZUODMNRITQFE66T46YWSFNMXVC7YDA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:39.149687765 (60h21m48.872250805s) [upload=57899]: Next returned nil segment Jul 26 23:57:39.149731879 (60h21m48.87229492s) [upload=57899]: Waiting for error group managing segments... Jul 26 23:57:39.455484025 (60h21m49.178047055s) [noise=true , address=79.127.205.246:4253 , piece_id=NATM45ZYE6H4RHXA4KNXW6CK7KQLPPRD4A7A6KOCJJWCSGYNJEEQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.473580770 (60h21m49.19614381s) [noise=true , address=79.127.205.245:4240 , piece_id=IFFO3URWPDROY7WJE3ZNRHLRU67NNJAD2LLQHOHYCRLFAYMOK6YQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.950821293 (60h21m49.673384333s) [noise=true , address=77.90.31.11:4585 , piece_id=56X44FTMER6FK647XF73E5EFU6NZQASTZH4YSK2Z77MQOJ635CUQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.958601049 (60h21m49.681164079s) [noise=true , address=77.90.31.16:4736 , piece_id=I2OIN67ZONMXFUFS4LFG6WJGH5S4E7VHW5DRANKCFCRJJ4EJZB2Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.982565981 (60h21m49.705129011s) [noise=true , address=79.127.219.38:4006 , piece_id=H76UQJJ47D6F6AUDYYDVUJX75PLRWT5PUL75YI2W7MR7KXGGSBLA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:40.160505642 (60h21m49.883068682s) [noise=true , address=64.49.13.138:4483 , piece_id=2OXH6RJVAIYXZKRO2DC7AMEAEYK4WINBV52HAQUK7WBPXYKXKZRA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:40.998834908 (60h21m50.721397948s) [noise=true , address=79.127.220.98:4421 , piece_id=6IJ3UDSZ3OONAIM4HJOH7FZMWZJDNL3JIZW2FQVF7G3GCJO5372A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:41.022368595 (60h21m50.744931675s) [noise=true , address=77.90.31.13:4629 , piece_id=ETIN7EO7P7GQ6O2RVKU5PRQAAKL47GCYD6AFEGUPI4VXILLPA6XA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:42.490649298 (60h21m52.213212358s) [noise=true , address=156.146.43.225:4448 , piece_id=SRHUMHAXD3JSLXF4QLOSSNWO2DO757HDGHQ3K73HG6XDLPVTIEYA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:46.089774242 (60h21m55.812337272s) [noise=true , address=45.91.116.14:4194 , piece_id=DD2FUOO3O6YAYJZ23TYGROGF3BEJDCOX7Z6ZFTLR32V4DY23IY3Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:58:36.954450190 (60h22m46.67701322s) [noise=true , address=104.36.82.56:4262 , piece_id=ULQXGOENYL6GJKQHHQUXZMXHZBZGBPLE7NGM7CF64ZELWJF3QGMA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954517127 (60h22m46.677080167s) [noise=true , address=104.36.82.46:4501 , piece_id=YESAAOIXCYHVCWOZXKWO4NAIEMJGB5TA4A3GDPCB333C6KOVYR3Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 188.166.225.170:41388->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954561182 (60h22m46.677124212s) [noise=true , address=104.36.82.47:4519 , piece_id=M6PHGB34J74SAE2ZYI44NJ3V4HNMFWS75VTDTH3FVP7R6RJCGKVA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: rpc: dial tcp 104.36.82.47:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954587533 (60h22m46.677150573s) [noise=true , address=104.36.82.53:4153 , piece_id=WLBTOA5UHXE2WFXGJTJ2AYUZXWXY6ZYQ7MN4ZR44B4XRJ7PXV47A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: rpc: dial tcp 104.36.82.53:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954610947 (60h22m46.677173988s) [noise=true , address=104.36.82.44:4894 , piece_id=YHXB52SOW4OPCIYBA5PQDGIBH3KLW3E2KWXGHXGHPHIP4WI7XKEA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954652166 (60h22m46.677215206s) [noise=true , address=104.36.82.55:4197 , piece_id=WNB5MAAIUHWHWBBY5EJE56WHRN2HT4EZLOLCVLWORCMYHPVOQ26A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954667125 (60h22m46.677230165s) [noise=true , address=77.90.31.10:4544 , piece_id=TIZSESFHKJXQXE2P2ATUSPYJHFP5VD7FM6D4YF6Z3UEUGEALGAQQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: rpc: dial tcp 77.90.31.10:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954685730 (60h22m46.67724877s) [noise=true , address=104.36.82.54:4174 , piece_id=2HDRVUUXQZ7MQCZQNLO2FXZUODMNRITQFE66T46YWSFNMXVC7YDA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: write tcp 188.166.225.170:56578->104.36.82.54:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954709726 (60h22m46.677272766s) [seg_pos={0 0}, upload=57899]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:58:37.182397114 (60h22m46.904960155s) [upload=57899]: Done uploading segments... Jul 26 23:58:37.182419267 (60h22m46.904982307s) [upload=57899]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230029483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX] duration: 30.232359179s FAIL (2025-07-26 23:58:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:44896->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:57:52.262108051 (60h22m1.984671091s) [seg_pos={0 0}, upload=57894]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:57:52.500328175 (60h22m2.222891205s) [upload=57894]: Done uploading segments... Jul 26 23:57:52.500353694 (60h22m2.222916734s) [upload=57894]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:57:53.376098356 (60h22m3.098661386s) [upload=57901]: Starting upload Jul 26 23:57:53.376203638 (60h22m3.098766668s) [upload=57901]: Done starting upload Jul 26 23:57:53.376274283 (60h22m3.098837324s) [upload=57901]: Uploading segments... Jul 26 23:57:53.376384194 (60h22m3.098947234s) [upload=57901]: Got next segment. Inline: false Jul 26 23:57:53.376410224 (60h22m3.098973264s) [upload=57901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:53.376414512 (60h22m3.098977552s) [upload=57901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:53.611631963 (60h22m3.334195003s) [seg_pos={0 0}, upload=57901]: Begin upload segment... Jul 26 23:57:53.611675807 (60h22m3.334238847s) [seg_pos={0 0}, upload=57901]: Done begin upload segment. Jul 26 23:57:53.611695715 (60h22m3.334258755s) [noise=true , address=45.91.116.5:4298 , piece_id=AFQ2QK24RUHRS6VXLFXSUXM7GOIUN2F3KFI7WZI2SRWB6UDFNNUA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.611725352 (60h22m3.334288392s) [noise=true , address=77.90.31.11:4553 , piece_id=QQ5PZQ5LHQLMFAYTPP6GGTYOIAWJIEZUL2DWCQ3IOZMCIM5XIBHA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612030927 (60h22m3.334593967s) [noise=true , address=77.90.31.16:4734 , piece_id=QBDKXFMXE3FF45VHMACJOOR5DN5LX6UICYBBENGEUWZRWHKNAWDQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612404340 (60h22m3.33496737s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XVCIODDL4TKYT26M5IVC5JHYSUVNZSYYQOBTXZN3MSCHR6LC2Q, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612560429 (60h22m3.335123469s) [noise=true , address=77.90.31.19:4841 , piece_id=ZPG2ATD2KCI5HKALKT67JACMLLZFYZPR4JMTB57FI7YXBAKQSHBQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612663056 (60h22m3.335226096s) [noise=true , address=77.90.31.19:4843 , piece_id=XV73NM2LA25COLBIJTVMSGEJJLI6SZYXS4KORLY4DUC7O6AFLGOQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612897173 (60h22m3.335460213s) [noise=true , address=104.36.82.7:4910 , piece_id=Z55Q4HDRMYWIMEDT7RQKYPY5T32GDW7OEYQ67UHDQKUF3TKRZTWQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.613188291 (60h22m3.335751331s) [noise=true , address=45.91.116.11:4118 , piece_id=AWVRI7A6EKV6XKOIYHHB5IUHHUX2HMTBG3B3Z3AQSMEXIOUER4RA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.613430855 (60h22m3.335993895s) [noise=true , address=45.91.116.5:4299 , piece_id=NOEHVVHQLTKKMOZFQN5AHGMVNWJR6J4W76SFQHWQH35ZH2XHUGJA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614210076 (60h22m3.336773116s) [noise=true , address=104.36.82.10:4078 , piece_id=UK33Z4KGYR257OJGV7YXQNVYGPE4CG4ZSBFSDVXTUW6OIT7YOPAA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614459212 (60h22m3.337022252s) [noise=true , address=45.91.116.9:4262 , piece_id=NBMSQ2GR47WBAOEIUZNIQE3I354YUEMNKC2CY6ZBDSPVM24Q2NHQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614778233 (60h22m3.337341273s) [noise=true , address=45.91.116.4:4939 , piece_id=7UQBD3YEUBVUMVSJELXDXS5HBSK6YJLK4SJW3XNWNGSNZNHT7FYQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615069850 (60h22m3.33763289s) [noise=true , address=77.90.31.18:4807 , piece_id=MEFRRCJG6COPKBQR2APTO5TCEOXP5O5MEJGREO4WORVFP6ZNF2IA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615309780 (60h22m3.33787282s) [noise=true , address=77.90.31.17:4770 , piece_id=IEJPIZFXE434BDQQZWWA3CKV6FDBLIUG4OBSZBHO342JZRY7FUEA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615542556 (60h22m3.338105596s) [noise=true , address=77.90.31.14:4661 , piece_id=FBKFSS4D4JOASPR4XHY2MKHXHD2HKPBIQESV4SBXTY75MGKDL3KA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615802703 (60h22m3.338365743s) [noise=true , address=45.91.116.8:4047 , piece_id=6A4WA7GCGF624XMCXVFEN7IT3TCQNZ7HT5VVTTB55AHR2VU4CLJQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:55.324701039 (60h22m5.047264109s) [upload=57901]: Next returned nil segment Jul 26 23:57:55.324723832 (60h22m5.047286863s) [upload=57901]: Waiting for error group managing segments... Jul 26 23:57:55.622628072 (60h22m5.345191102s) [noise=true , address=45.91.116.5:4298 , piece_id=AFQ2QK24RUHRS6VXLFXSUXM7GOIUN2F3KFI7WZI2SRWB6UDFNNUA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.641771479 (60h22m5.364334519s) [noise=true , address=45.91.116.4:4939 , piece_id=7UQBD3YEUBVUMVSJELXDXS5HBSK6YJLK4SJW3XNWNGSNZNHT7FYQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.698429273 (60h22m5.420992313s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XVCIODDL4TKYT26M5IVC5JHYSUVNZSYYQOBTXZN3MSCHR6LC2Q, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.726265291 (60h22m5.448828331s) [noise=true , address=45.91.116.5:4299 , piece_id=NOEHVVHQLTKKMOZFQN5AHGMVNWJR6J4W76SFQHWQH35ZH2XHUGJA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.740361332 (60h22m5.462924372s) [noise=true , address=77.90.31.14:4661 , piece_id=FBKFSS4D4JOASPR4XHY2MKHXHD2HKPBIQESV4SBXTY75MGKDL3KA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.767343256 (60h22m5.489906296s) [noise=true , address=45.91.116.11:4118 , piece_id=AWVRI7A6EKV6XKOIYHHB5IUHHUX2HMTBG3B3Z3AQSMEXIOUER4RA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.798300466 (60h22m5.520863506s) [noise=true , address=77.90.31.11:4553 , piece_id=QQ5PZQ5LHQLMFAYTPP6GGTYOIAWJIEZUL2DWCQ3IOZMCIM5XIBHA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.801780304 (60h22m5.524343344s) [noise=true , address=45.91.116.8:4047 , piece_id=6A4WA7GCGF624XMCXVFEN7IT3TCQNZ7HT5VVTTB55AHR2VU4CLJQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.803751445 (60h22m5.526314485s) [noise=true , address=77.90.31.16:4734 , piece_id=QBDKXFMXE3FF45VHMACJOOR5DN5LX6UICYBBENGEUWZRWHKNAWDQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.807686584 (60h22m5.530249624s) [noise=true , address=45.91.116.9:4262 , piece_id=NBMSQ2GR47WBAOEIUZNIQE3I354YUEMNKC2CY6ZBDSPVM24Q2NHQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:57.091454785 (60h22m6.814017825s) [noise=true , address=77.90.31.19:4841 , piece_id=ZPG2ATD2KCI5HKALKT67JACMLLZFYZPR4JMTB57FI7YXBAKQSHBQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:57.562405322 (60h22m7.284968362s) [noise=true , address=77.90.31.19:4843 , piece_id=XV73NM2LA25COLBIJTVMSGEJJLI6SZYXS4KORLY4DUC7O6AFLGOQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:58:22.719662701 (60h22m32.442225731s) [noise=true , address=77.90.31.17:4770 , piece_id=IEJPIZFXE434BDQQZWWA3CKV6FDBLIUG4OBSZBHO342JZRY7FUEA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:46326->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719727925 (60h22m32.442290955s) [noise=true , address=104.36.82.10:4078 , piece_id=UK33Z4KGYR257OJGV7YXQNVYGPE4CG4ZSBFSDVXTUW6OIT7YOPAA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:34810->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719778071 (60h22m32.442341101s) [noise=true , address=77.90.31.18:4807 , piece_id=MEFRRCJG6COPKBQR2APTO5TCEOXP5O5MEJGREO4WORVFP6ZNF2IA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection Jul 26 23:58:22.719793410 (60h22m32.44235645s) [noise=true , address=104.36.82.7:4910 , piece_id=Z55Q4HDRMYWIMEDT7RQKYPY5T32GDW7OEYQ67UHDQKUF3TKRZTWQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:48550->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719819740 (60h22m32.44238278s) [seg_pos={0 0}, upload=57901]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:58:22.950197325 (60h22m32.672760365s) [upload=57901]: Done uploading segments... Jul 26 23:58:22.950224778 (60h22m32.672787808s) [upload=57901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232373095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX..] duration: 13.380103738s FAIL (2025-07-26 23:57:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.380124128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.X] duration: 30.240920465s FAIL (2025-07-26 23:57:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e starting upload Jul 26 23:57:22.900804744 (60h21m32.623367784s) [upload=57894]: Uploading segments... Jul 26 23:57:22.900899145 (60h21m32.623462185s) [upload=57894]: Got next segment. Inline: false Jul 26 23:57:22.900941105 (60h21m32.623504145s) [upload=57894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:22.900945835 (60h21m32.623508865s) [upload=57894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:23.132328870 (60h21m32.85489191s) [seg_pos={0 0}, upload=57894]: Begin upload segment... Jul 26 23:57:23.132368847 (60h21m32.854931887s) [seg_pos={0 0}, upload=57894]: Done begin upload segment. Jul 26 23:57:23.132388755 (60h21m32.854951795s) [noise=true , address=104.36.82.8:4972 , piece_id=ESU44TKC6WEYX26477UNQY256ZXLKQ2DBAORGV6UOZEBNRDMB5AA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.132438159 (60h21m32.855001199s) [noise=true , address=77.90.31.19:4843 , piece_id=AQPCT6DM2C74MNM3GEH5VMJQQQYPE6EDOIFZLLLHVSLAKYEORFOQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.132881618 (60h21m32.855444658s) [noise=true , address=104.36.82.3:4765 , piece_id=FJBVENVGZFVVM7ZQTQZPNANFTEWR3IWKGOQ26KB3TEXQTV2MWDXQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133038177 (60h21m32.855601217s) [noise=true , address=77.90.31.18:4805 , piece_id=6T7UHMAGFCHGAZCMJNQXWKXSBLKWFRH4Y2CCLTEO5QMBFHNVQQDA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133175911 (60h21m32.855738951s) [noise=true , address=45.91.116.7:4010 , piece_id=UMX7Z456LBAXBQ5MM3VWX6QGNWBIOO73DPPMH4OC5C2KUF22JU7A, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133307493 (60h21m32.855870533s) [noise=true , address=104.36.82.9:4007 , piece_id=W2RSP2CTIHTQEECF72OAOTGKL4J6XHEJSKYDBQ7ZAJM62NNL6SSA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133455315 (60h21m32.856018355s) [noise=true , address=77.90.31.21:4878 , piece_id=JGQS42AAEIMXJLHLDXM7LT6M7WFQN3XDXGAPN2G52TWVAUFGWZFA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133733377 (60h21m32.856296417s) [noise=true , address=45.91.116.7:4009 , piece_id=K5NU7FUKXMMCPANCX6UGFDNVBLXLL2HZHYWV5AZEZXOA3CZEPGKQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133781500 (60h21m32.85634454s) [noise=true , address=104.36.82.4:4802 , piece_id=N6IKNZJTWB3BPPYKB7JU4R6NBDHZ2VWQXGD7G2BMZKB4QW6U6JYA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134005458 (60h21m32.856568498s) [noise=true , address=45.91.116.11:4119 , piece_id=OR3O5ZPNKNAJDMBD5362SF4Y7WCOKTPP32W55CABPV73LVBW7BEA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134062096 (60h21m32.856625136s) [noise=true , address=45.91.116.12:4226 , piece_id=7UMQUC5FEMSOZQOAP3M752HC6MCNCA7ZGUMJD2PUJ2LCVBSW4DCQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134278200 (60h21m32.85684123s) [noise=true , address=77.90.31.11:4555 , piece_id=LNZ6J2JAVZDZUW2QDWNO3SJVY267ZQADYNPONOGSFSJC4IIMFIEA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134308598 (60h21m32.856871638s) [noise=true , address=45.91.116.10:4083 , piece_id=2TGO6QIKBTMOHBFO7TZWEPABGPVM2FGGB3HHLZA3AZ46T6T6WTLQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134523058 (60h21m32.857086099s) [noise=true , address=77.90.31.19:4841 , piece_id=AZMC5AMIK5WH3CKGSKVOT7QPCWRC55HS5HOPPUJNJUXKF2327AEQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134724854 (60h21m32.857287884s) [noise=true , address=104.36.82.7:4909 , piece_id=XUDABXU6CK3U7VALBAVEJ56OPE34Z2S2QPRDUCZBMAQVEIFWIBUQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134922111 (60h21m32.857485152s) [noise=true , address=104.36.82.16:4295 , piece_id=7X2HWJUJO5M5QYBC7NBDY2RRTMGHD7BYPN3N6CGJUJ3EXGOLPYDA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:24.874325773 (60h21m34.596888803s) [upload=57894]: Next returned nil segment Jul 26 23:57:24.874342895 (60h21m34.596905935s) [upload=57894]: Waiting for error group managing segments... Jul 26 23:57:25.158541383 (60h21m34.881104423s) [noise=true , address=45.91.116.7:4009 , piece_id=K5NU7FUKXMMCPANCX6UGFDNVBLXLL2HZHYWV5AZEZXOA3CZEPGKQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.229145159 (60h21m34.951708199s) [noise=true , address=77.90.31.11:4555 , piece_id=LNZ6J2JAVZDZUW2QDWNO3SJVY267ZQADYNPONOGSFSJC4IIMFIEA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.235963203 (60h21m34.958526233s) [noise=true , address=45.91.116.12:4226 , piece_id=7UMQUC5FEMSOZQOAP3M752HC6MCNCA7ZGUMJD2PUJ2LCVBSW4DCQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.325026678 (60h21m35.047589718s) [noise=true , address=45.91.116.10:4083 , piece_id=2TGO6QIKBTMOHBFO7TZWEPABGPVM2FGGB3HHLZA3AZ46T6T6WTLQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.374807515 (60h21m35.097370555s) [noise=true , address=45.91.116.7:4010 , piece_id=UMX7Z456LBAXBQ5MM3VWX6QGNWBIOO73DPPMH4OC5C2KUF22JU7A, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.395860095 (60h21m35.118423135s) [noise=true , address=45.91.116.11:4119 , piece_id=OR3O5ZPNKNAJDMBD5362SF4Y7WCOKTPP32W55CABPV73LVBW7BEA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:26.596637682 (60h21m36.319200722s) [noise=true , address=77.90.31.19:4843 , piece_id=AQPCT6DM2C74MNM3GEH5VMJQQQYPE6EDOIFZLLLHVSLAKYEORFOQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:26.761740593 (60h21m36.484303633s) [noise=true , address=77.90.31.19:4841 , piece_id=AZMC5AMIK5WH3CKGSKVOT7QPCWRC55HS5HOPPUJNJUXKF2327AEQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:52.261458277 (60h22m1.984021317s) [noise=true , address=77.90.31.21:4878 , piece_id=JGQS42AAEIMXJLHLDXM7LT6M7WFQN3XDXGAPN2G52TWVAUFGWZFA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:52.261614286 (60h22m1.984177326s) [noise=true , address=104.36.82.7:4909 , piece_id=XUDABXU6CK3U7VALBAVEJ56OPE34Z2S2QPRDUCZBMAQVEIFWIBUQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:46122->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261863423 (60h22m1.984426463s) [noise=true , address=104.36.82.9:4007 , piece_id=W2RSP2CTIHTQEECF72OAOTGKL4J6XHEJSKYDBQ7ZAJM62NNL6SSA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:46614->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261895494 (60h22m1.984458534s) [noise=true , address=104.36.82.16:4295 , piece_id=7X2HWJUJO5M5QYBC7NBDY2RRTMGHD7BYPN3N6CGJUJ3EXGOLPYDA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:35576->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261926092 (60h22m1.984489123s) [noise=true , address=104.36.82.3:4765 , piece_id=FJBVENVGZFVVM7ZQTQZPNANFTEWR3IWKGOQ26KB3TEXQTV2MWDXQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:54596->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261957924 (60h22m1.984520964s) [noise=true , address=104.36.82.8:4972 , piece_id=ESU44TKC6WEYX26477UNQY256ZXLKQ2DBAORGV6UOZEBNRDMB5AA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:36806->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261993041 (60h22m1.984556081s) [noise=true , address=104.36.82.4:4802 , piece_id=N6IKNZJTWB3BPPYKB7JU4R6NBDHZ2VWQXGD7G2BMZKB4QW6U6JYA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:50144->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.262093873 (60h22m1.984656903s) [noise=true , address=77.90.31.18:4805 , piece_id=6T7UHMAGFCHGAZCMJNQXWKXSBLKWFRH4Y2CCLTEO5QMBFHNVQQDA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:44896->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:57:52.262108051 (60h22m1.984671091s) [seg_pos={0 0}, upload=57894]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:57:52.500328175 (60h22m2.222891205s) [upload=57894]: Done uploading segments... Jul 26 23:57:52.500353694 (60h22m2.222916734s) [upload=57894]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.24094465s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)