Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..] duration: 29.413031147s FAIL (2025-07-27 00:01:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.413046005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXX] duration: 30.235163086s FAIL (2025-07-27 00:01:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments... Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:00:34.625856171 (60h24m44.348419211s) [upload=57946]: Starting upload Jul 27 00:00:34.625958327 (60h24m44.348521407s) [upload=57946]: Done starting upload Jul 27 00:00:34.626040784 (60h24m44.348603824s) [upload=57946]: Uploading segments... Jul 27 00:00:34.626264823 (60h24m44.348827863s) [upload=57946]: Got next segment. Inline: false Jul 27 00:00:34.626311623 (60h24m44.348874663s) [upload=57946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:00:34.626317774 (60h24m44.348880844s) [upload=57946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:00:34.865594065 (60h24m44.588157106s) [seg_pos={0 0}, upload=57946]: Begin upload segment... Jul 27 00:00:34.865647087 (60h24m44.588210127s) [seg_pos={0 0}, upload=57946]: Done begin upload segment. Jul 27 00:00:34.865670111 (60h24m44.588233141s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.865708626 (60h24m44.588271666s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866075106 (60h24m44.588638146s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866256885 (60h24m44.588819925s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866404937 (60h24m44.588967978s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866648443 (60h24m44.589211484s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866667950 (60h24m44.58923099s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.866930773 (60h24m44.589493813s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867221260 (60h24m44.58978429s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867547112 (60h24m44.590110152s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.867777634 (60h24m44.590340674s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868010669 (60h24m44.590573699s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868350270 (60h24m44.59091331s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868577584 (60h24m44.591140624s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.868844886 (60h24m44.591407926s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:34.869181019 (60h24m44.591744049s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Uploading piece... Jul 27 00:00:36.571647152 (60h24m46.294210192s) [upload=57946]: Next returned nil segment Jul 27 00:00:36.571683281 (60h24m46.294246331s) [upload=57946]: Waiting for error group managing segments... Jul 27 00:00:36.869992438 (60h24m46.592555478s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.887108205 (60h24m46.609671245s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.907036524 (60h24m46.629599574s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.958513032 (60h24m46.681076072s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.973207827 (60h24m46.695770867s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.974133909 (60h24m46.696696949s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.977135692 (60h24m46.699698732s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:36.978503389 (60h24m46.701066429s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.016312929 (60h24m46.738875969s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.021475726 (60h24m46.744038766s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:00:37.036167487 (60h24m46.758730527s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil> Jul 27 00:01:03.968881072 (60h25m13.691444112s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968922361 (60h25m13.691485401s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:53216->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968928994 (60h25m13.691492034s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:37916->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968952910 (60h25m13.69151595s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968971746 (60h25m13.691534786s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments... Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.235193273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X...] duration: 12.246817958s FAIL (2025-07-27 00:01:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.246844077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.99403249s FAIL (2025-07-27 00:00:50 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.99405258s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.] duration: 50.489387343s FAIL (2025-07-27 00:00:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.489408945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.] duration: 5.889514613s FAIL (2025-07-27 00:00:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.889541043s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236620041s FAIL (2025-07-27 00:00:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Jul 26 23:59:42.179886117 (60h23m51.902449157s) [seg_pos={0 0}, upload=57933]: Done begin upload segment. Jul 26 23:59:42.179910323 (60h23m51.902473363s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.179916154 (60h23m51.902479184s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.179962733 (60h23m51.902525763s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180351267 (60h23m51.902914327s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180808632 (60h23m51.903371672s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.180835734 (60h23m51.903398774s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181092325 (60h23m51.903655355s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181114728 (60h23m51.903677758s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181342373 (60h23m51.903905403s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181360328 (60h23m51.903923368s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181588113 (60h23m51.904151153s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.181851127 (60h23m51.904414167s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182136052 (60h23m51.904699082s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182437568 (60h23m51.905000598s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182650676 (60h23m51.905213716s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.182935721 (60h23m51.905498761s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.183193584 (60h23m51.905756614s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:42.183391925 (60h23m51.905954965s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Uploading piece... Jul 26 23:59:43.285348328 (60h23m53.007911368s) [upload=57933]: Next returned nil segment Jul 26 23:59:43.285391911 (60h23m53.007954951s) [upload=57933]: Waiting for error group managing segments... Jul 26 23:59:43.598226841 (60h23m53.320789891s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.601115758 (60h23m53.323678798s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.642269155 (60h23m53.364832195s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.668410962 (60h23m53.390974012s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.708835476 (60h23m53.431398516s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:43.718426806 (60h23m53.440989846s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.097208810 (60h23m53.81977184s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.100794370 (60h23m53.823357411s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.132912180 (60h23m53.85547522s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 26 23:59:44.491805251 (60h23m54.214368291s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 27 00:00:18.017461556 (60h24m27.740024606s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil> Jul 27 00:00:41.095307490 (60h24m50.81787053s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095364931 (60h24m50.817927971s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095404997 (60h24m50.817968037s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:40312->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095434554 (60h24m50.817997594s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095483527 (60h24m50.818046578s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:4483): protocol: rpc: dial tcp 104.36.82.21:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095546067 (60h24m50.818109107s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1yyUzkigTS9m3bDrZtAFL7tciNxpiCNswBSfBqXn1UW83GMLwi, address:104.36.82.10:4091): protocol: write tcp 188.166.225.170:53172->104.36.82.10:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095578399 (60h24m50.818141439s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: read tcp 188.166.225.170:48104->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:41.095609017 (60h24m50.818172057s) [seg_pos={0 0}, upload=57933]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:00:41.330560987 (60h24m51.053124027s) [upload=57933]: Done uploading segments... Jul 27 00:00:41.330588720 (60h24m51.05315176s) [upload=57933]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.236633226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.949580404s FAIL (2025-07-27 00:00:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.949610942s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXX] duration: 30.2382672s FAIL (2025-07-27 00:00:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== [seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments... Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:59:53.265389883 (60h24m2.987952923s) [upload=57935]: Starting upload Jul 26 23:59:53.265451531 (60h24m2.988014571s) [upload=57935]: Done starting upload Jul 26 23:59:53.265547215 (60h24m2.988110255s) [upload=57935]: Uploading segments... Jul 26 23:59:53.265665881 (60h24m2.988228921s) [upload=57935]: Got next segment. Inline: false Jul 26 23:59:53.265695618 (60h24m2.988258648s) [upload=57935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:53.265700678 (60h24m2.988263718s) [upload=57935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:53.500569526 (60h24m3.223132566s) [seg_pos={0 0}, upload=57935]: Begin upload segment... Jul 26 23:59:53.500614643 (60h24m3.223177673s) [seg_pos={0 0}, upload=57935]: Done begin upload segment. Jul 26 23:59:53.500637407 (60h24m3.223200447s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.500687723 (60h24m3.223250763s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.500991074 (60h24m3.223554114s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501094200 (60h24m3.22365724s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501236714 (60h24m3.223799744s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501365379 (60h24m3.223928419s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501482303 (60h24m3.224045343s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501645385 (60h24m3.224208425s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501744114 (60h24m3.224307154s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.501899962 (60h24m3.224463003s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502000194 (60h24m3.224563234s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502145102 (60h24m3.224708142s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502305698 (60h24m3.224868738s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502386763 (60h24m3.224949793s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502540347 (60h24m3.225103377s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:53.502817999 (60h24m3.225381039s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Uploading piece... Jul 26 23:59:55.245077228 (60h24m4.967640258s) [upload=57935]: Next returned nil segment Jul 26 23:59:55.245105442 (60h24m4.967668482s) [upload=57935]: Waiting for error group managing segments... Jul 26 23:59:55.521486724 (60h24m5.244049764s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.535621258 (60h24m5.258184288s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.590385597 (60h24m5.312948627s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.596108576 (60h24m5.318671616s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.596498993 (60h24m5.319062023s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.607949671 (60h24m5.330512711s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.614678556 (60h24m5.337241596s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.632027378 (60h24m5.354590419s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.675652446 (60h24m5.398215486s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 26 23:59:55.807225580 (60h24m5.529788621s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil> Jul 27 00:00:22.622487945 (60h24m32.345050985s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection Jul 27 00:00:22.622567417 (60h24m32.345130467s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622596592 (60h24m32.345159633s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622627972 (60h24m32.345191012s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622653822 (60h24m32.345216862s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622694220 (60h24m32.34525726s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments... Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.238292498s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X..] duration: 5.917415052s FAIL (2025-07-27 00:00:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.917451352s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX] duration: 30.244538304s FAIL (2025-07-27 00:00:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.244561219s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X] duration: 1m0.00115051s FAIL (2025-07-26 23:59:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001161703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXX] duration: 30.239655162s FAIL (2025-07-26 23:59:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 6 23:59:18.755564131 (60h23m28.478127171s) [upload=57925]: Starting upload Jul 26 23:59:18.755617964 (60h23m28.478181004s) [upload=57925]: Done starting upload Jul 26 23:59:18.755706744 (60h23m28.478269775s) [upload=57925]: Uploading segments... Jul 26 23:59:18.755814431 (60h23m28.478377471s) [upload=57925]: Got next segment. Inline: false Jul 26 23:59:18.755857272 (60h23m28.478420312s) [upload=57925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:18.755863173 (60h23m28.478426223s) [upload=57925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:18.989710817 (60h23m28.712273857s) [seg_pos={0 0}, upload=57925]: Begin upload segment... Jul 26 23:59:18.989748790 (60h23m28.71231183s) [seg_pos={0 0}, upload=57925]: Done begin upload segment. Jul 26 23:59:18.989766403 (60h23m28.712329443s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.989850995 (60h23m28.712414035s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990095784 (60h23m28.712658824s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990392171 (60h23m28.712955211s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990510397 (60h23m28.713073437s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990643101 (60h23m28.713206131s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.990975377 (60h23m28.713538417s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991103472 (60h23m28.713666542s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991392835 (60h23m28.713955875s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991486916 (60h23m28.714049956s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991739719 (60h23m28.714302749s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991790737 (60h23m28.714353777s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.991946946 (60h23m28.714509986s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992186614 (60h23m28.714749644s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992254975 (60h23m28.714818015s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:18.992672093 (60h23m28.715235133s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Uploading piece... Jul 26 23:59:20.729860921 (60h23m30.452423961s) [upload=57925]: Next returned nil segment Jul 26 23:59:20.729875970 (60h23m30.45243901s) [upload=57925]: Waiting for error group managing segments... Jul 26 23:59:21.008316574 (60h23m30.730879614s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.008638079 (60h23m30.731201119s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.024163012 (60h23m30.746726052s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.160487698 (60h23m30.883050738s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.162286621 (60h23m30.884849661s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:21.461775184 (60h23m31.184338224s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:22.102625777 (60h23m31.825188817s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:22.398677264 (60h23m32.121240304s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil> Jul 26 23:59:48.106089545 (60h23m57.828652585s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106139840 (60h23m57.82870288s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106225273 (60h23m57.828788313s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:35526->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106281591 (60h23m57.828844631s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106335895 (60h23m57.828898925s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106366715 (60h23m57.828929755s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:42844->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106414085 (60h23m57.828977115s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106446386 (60h23m57.829009426s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:48.106468108 (60h23m57.829031148s) [seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments... Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.239673657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233995905s FAIL (2025-07-26 23:59:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.727576947 (60h22m51.450139977s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728137028 (60h22m51.450700068s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728432825 (60h22m51.450995865s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728479454 (60h22m51.451042494s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728761574 (60h22m51.451324614s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.728956907 (60h22m51.451519947s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729045346 (60h22m51.451608386s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729528752 (60h22m51.452091802s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.729779631 (60h22m51.452342661s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730008299 (60h22m51.452571339s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730341586 (60h22m51.452904626s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.730668923 (60h22m51.453231963s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.731030434 (60h22m51.453593474s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.731448755 (60h22m51.454011785s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.732112174 (60h22m51.454675214s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.732720969 (60h22m51.455284019s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:41.733091287 (60h22m51.455654327s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Uploading piece... Jul 26 23:58:42.838834658 (60h22m52.561397688s) [upload=57914]: Next returned nil segment Jul 26 23:58:42.838851310 (60h22m52.56141435s) [upload=57914]: Waiting for error group managing segments... Jul 26 23:58:43.129009261 (60h22m52.851572311s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.138456927 (60h22m52.861019957s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.163842056 (60h22m52.886405096s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.583641918 (60h22m53.306204958s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.724061759 (60h22m53.446624799s) [noise=true , address=79.127.219.38:4010 , piece_id=XE3WNW4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:43.801955510 (60h22m53.52451855s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:44.316271165 (60h22m54.038834246s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:58:45.515232647 (60h22m55.237795707s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:08.772393964 (60h23m18.494957004s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:23.999032802 (60h23m33.721595842s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil> Jul 26 23:59:40.631535807 (60h23m50.354098847s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op Jul 26 23:59:40.631613436 (60h23m50.354176476s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631773583 (60h23m50.354336623s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: write tcp 188.166.225.170:58646->104.36.82.53:4144: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631808780 (60h23m50.35437182s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: read tcp 188.166.225.170:44574->77.90.31.10:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631841593 (60h23m50.354404633s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:40786->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631884745 (60h23m50.354447785s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: rpc: dial tcp 104.36.82.56:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631949940 (60h23m50.35451298s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 188.166.225.170:59158->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.631997791 (60h23m50.354560821s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:40.632018481 (60h23m50.354581521s) [seg_pos={0 0}, upload=57914]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:59:40.863499046 (60h23m50.586062086s) [upload=57914]: Done uploading segments... Jul 26 23:59:40.863526769 (60h23m50.586089799s) [upload=57914]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234016886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X.] duration: 5.91084676s FAIL (2025-07-26 23:59:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.91087298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.855632713s FAIL (2025-07-26 23:59:36 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.855661519s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration: 30.2216459s FAIL (2025-07-26 23:59:12 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 30.221660339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXX] duration: 30.232021331s FAIL (2025-07-26 23:59:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 1]: Done begin upload segment. Jul 26 23:58:37.367348346 (60h22m47.089911386s) [noise=true , address=104.36.82.15:4258 , piece_id=KJ4L6EGODVNLLI4Y5DOZLAGERZWCESEL32PRIDZAM2ZOHVKYBQTA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367354979 (60h22m47.089918019s) [noise=true , address=104.36.82.5:4839 , piece_id=VLZIP4UTTVBNFM5ELQBBMEPUWOOV5YRIZYDSELWQMGH7TTZYBPBQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367651737 (60h22m47.090214767s) [noise=true , address=77.90.31.16:4735 , piece_id=7YZHQWTT55WX7MFHC7RBZAPLK4V62DLHL5JMEYO5EEETJ4C4FEVQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367814779 (60h22m47.090377819s) [noise=true , address=77.90.31.18:4805 , piece_id=URT7OEWA2FBA2IXTSNJMGRJBICUYS6EC22RZAM5K26XBSQN62XEQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.367941231 (60h22m47.090504271s) [noise=true , address=77.90.31.15:4698 , piece_id=JZJHPXR2TWAMUNEW6UL3UACSLBQRU3W4ELRGRA75Y43NUG5ZVFHQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368222088 (60h22m47.090785128s) [noise=true , address=77.90.31.21:4878 , piece_id=635AR4P2ZZRAS7MJ5ZKH4AIJSWFDY25CW3AKXEKLLU7XHFZTLZ4Q, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368453270 (60h22m47.09101631s) [noise=true , address=45.91.116.5:4299 , piece_id=VGWYIVCJW3U77BD6W2AELSY5DHHY56UHFLDHH23QNWKUQEY7WBTQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368674463 (60h22m47.091237503s) [noise=true , address=77.90.31.21:4879 , piece_id=IKN6NHWPFKD7VSYI76436BLJRMG7DSEHBQMLEBNKAEP5YA7LJOOA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.368899384 (60h22m47.091462414s) [noise=true , address=45.91.116.6:4974 , piece_id=G7XXUR3ZAF2TMELMYKKLZGSRY26FJLZM5C6IIYUDFZ2UBUEWZ4SQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.369267328 (60h22m47.091830358s) [noise=true , address=104.36.82.3:4765 , piece_id=DGSSUPGJZGRLFKFRLGVKXXHUQ4KU3ANOYIKRRJAFWXGECUWL52AQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.369724183 (60h22m47.092287213s) [noise=true , address=104.36.82.12:4149 , piece_id=JMMSTQ36PQV7EYK7Y3XL3ES34VGOJP7UOFGVZKJHHY7XAORWGEAA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370241081 (60h22m47.092804121s) [noise=true , address=45.91.116.4:4937 , piece_id=5HO4BGY7L5T5MXZA2QHP6IAC5U3UQQLI2XXAOBITTX64XT5RJEEQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370447636 (60h22m47.093010676s) [noise=true , address=45.91.116.13:4153 , piece_id=MYPRMK4SWEZJHJVRIVWE257GCEFB6GXT2KBKSDRZUKX73QADOMYQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370637029 (60h22m47.093200069s) [noise=true , address=104.36.82.9:4007 , piece_id=TH2JYY4B3SOKN6T3WJUPHK25OYRDIEMIY3YMYMXHYVOFQNDUK4TQ, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.370825760 (60h22m47.0933888s) [noise=true , address=104.36.82.11:4115 , piece_id=2HYDVHINWBEMOB7SG3WGEQVRSNXKLA4KWKB6JF65ZFMF7VBMGAPA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:37.371029900 (60h22m47.09359294s) [noise=true , address=77.90.31.16:4734 , piece_id=CICRVUAX6JG2H7GRCTFSHRUB3SD3LQZ3XC2N6PTB4THKRP2CB3GA, seg_pos={0 0}, upload=57911]: Uploading piece... Jul 26 23:58:59.233070760 (60h23m8.95563381s) [upload=57911]: Next returned nil segment Jul 26 23:58:59.233101228 (60h23m8.955664288s) [upload=57911]: Waiting for error group managing segments... Jul 26 23:58:59.569938991 (60h23m9.292502031s) [noise=true , address=45.91.116.5:4299 , piece_id=VGWYIVCJW3U77BD6W2AELSY5DHHY56UHFLDHH23QNWKUQEY7WBTQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:00.216394257 (60h23m9.938957297s) [noise=true , address=45.91.116.4:4937 , piece_id=5HO4BGY7L5T5MXZA2QHP6IAC5U3UQQLI2XXAOBITTX64XT5RJEEQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:00.738752451 (60h23m10.461315491s) [noise=true , address=45.91.116.6:4974 , piece_id=G7XXUR3ZAF2TMELMYKKLZGSRY26FJLZM5C6IIYUDFZ2UBUEWZ4SQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:02.413157868 (60h23m12.135720908s) [noise=true , address=77.90.31.15:4698 , piece_id=JZJHPXR2TWAMUNEW6UL3UACSLBQRU3W4ELRGRA75Y43NUG5ZVFHQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:03.173280099 (60h23m12.895843139s) [noise=true , address=45.91.116.13:4153 , piece_id=MYPRMK4SWEZJHJVRIVWE257GCEFB6GXT2KBKSDRZUKX73QADOMYQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:05.916467888 (60h23m15.639030928s) [noise=true , address=77.90.31.18:4805 , piece_id=URT7OEWA2FBA2IXTSNJMGRJBICUYS6EC22RZAM5K26XBSQN62XEQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: <nil> Jul 26 23:59:06.490595672 (60h23m16.213158702s) [noise=true , address=77.90.31.16:4735 , piece_id=7YZHQWTT55WX7MFHC7RBZAPLK4V62DLHL5JMEYO5EEETJ4C4FEVQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52688->77.90.31.16:4735: use of closed network connection Jul 26 23:59:06.490638293 (60h23m16.213201333s) [noise=true , address=77.90.31.21:4878 , piece_id=635AR4P2ZZRAS7MJ5ZKH4AIJSWFDY25CW3AKXEKLLU7XHFZTLZ4Q, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490670275 (60h23m16.213233305s) [noise=true , address=104.36.82.3:4765 , piece_id=DGSSUPGJZGRLFKFRLGVKXXHUQ4KU3ANOYIKRRJAFWXGECUWL52AQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490716222 (60h23m16.213279262s) [noise=true , address=104.36.82.11:4115 , piece_id=2HYDVHINWBEMOB7SG3WGEQVRSNXKLA4KWKB6JF65ZFMF7VBMGAPA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490745899 (60h23m16.213308939s) [noise=true , address=104.36.82.15:4258 , piece_id=KJ4L6EGODVNLLI4Y5DOZLAGERZWCESEL32PRIDZAM2ZOHVKYBQTA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490787809 (60h23m16.213350849s) [noise=true , address=104.36.82.12:4149 , piece_id=JMMSTQ36PQV7EYK7Y3XL3ES34VGOJP7UOFGVZKJHHY7XAORWGEAA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53152->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490795424 (60h23m16.213358464s) [noise=true , address=104.36.82.5:4839 , piece_id=VLZIP4UTTVBNFM5ELQBBMEPUWOOV5YRIZYDSELWQMGH7TTZYBPBQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490832855 (60h23m16.213395895s) [noise=true , address=77.90.31.21:4879 , piece_id=IKN6NHWPFKD7VSYI76436BLJRMG7DSEHBQMLEBNKAEP5YA7LJOOA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490841562 (60h23m16.213404602s) [noise=true , address=104.36.82.9:4007 , piece_id=TH2JYY4B3SOKN6T3WJUPHK25OYRDIEMIY3YMYMXHYVOFQNDUK4TQ, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:06.490848946 (60h23m16.213411986s) [noise=true , address=77.90.31.16:4734 , piece_id=CICRVUAX6JG2H7GRCTFSHRUB3SD3LQZ3XC2N6PTB4THKRP2CB3GA, seg_pos={0 0}, upload=57911]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.16:4734: use of closed network connection Jul 26 23:59:06.490862603 (60h23m16.213425643s) [seg_pos={0 0}, upload=57911]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:59:06.720485622 (60h23m16.443048662s) [upload=57911]: Done uploading segments... Jul 26 23:59:06.720515039 (60h23m16.443078079s) [upload=57911]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.232044888s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X] duration: 30.000483269s FAIL (2025-07-26 23:59:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000506663s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX...] duration: 52.323428275s FAIL (2025-07-26 23:58:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.323448635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.854377712s FAIL (2025-07-26 23:58:45 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.854401578s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23001768s FAIL (2025-07-26 23:58:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 219.38:4006 , piece_id=H76UQJJ47D6F6AUDYYDVUJX75PLRWT5PUL75YI2W7MR7KXGGSBLA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.045959812 (60h21m47.768522852s) [noise=true , address=77.90.31.13:4629 , piece_id=ETIN7EO7P7GQ6O2RVKU5PRQAAKL47GCYD6AFEGUPI4VXILLPA6XA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046315172 (60h21m47.768878212s) [noise=true , address=79.127.220.98:4421 , piece_id=6IJ3UDSZ3OONAIM4HJOH7FZMWZJDNL3JIZW2FQVF7G3GCJO5372A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046427216 (60h21m47.768990256s) [noise=true , address=104.36.82.47:4519 , piece_id=M6PHGB34J74SAE2ZYI44NJ3V4HNMFWS75VTDTH3FVP7R6RJCGKVA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046666314 (60h21m47.769229344s) [noise=true , address=156.146.43.225:4448 , piece_id=SRHUMHAXD3JSLXF4QLOSSNWO2DO757HDGHQ3K73HG6XDLPVTIEYA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046714566 (60h21m47.769277596s) [noise=true , address=64.49.13.138:4483 , piece_id=2OXH6RJVAIYXZKRO2DC7AMEAEYK4WINBV52HAQUK7WBPXYKXKZRA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.046939245 (60h21m47.769502286s) [noise=true , address=104.36.82.55:4197 , piece_id=WNB5MAAIUHWHWBBY5EJE56WHRN2HT4EZLOLCVLWORCMYHPVOQ26A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047020601 (60h21m47.769583641s) [noise=true , address=77.90.31.11:4585 , piece_id=56X44FTMER6FK647XF73E5EFU6NZQASTZH4YSK2Z77MQOJ635CUQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047209683 (60h21m47.769772713s) [noise=true , address=104.36.82.44:4894 , piece_id=YHXB52SOW4OPCIYBA5PQDGIBH3KLW3E2KWXGHXGHPHIP4WI7XKEA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047435766 (60h21m47.769998806s) [noise=true , address=79.127.205.246:4253 , piece_id=NATM45ZYE6H4RHXA4KNXW6CK7KQLPPRD4A7A6KOCJJWCSGYNJEEQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047656819 (60h21m47.770219859s) [noise=true , address=79.127.205.245:4240 , piece_id=IFFO3URWPDROY7WJE3ZNRHLRU67NNJAD2LLQHOHYCRLFAYMOK6YQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.047876009 (60h21m47.770439049s) [noise=true , address=104.36.82.53:4153 , piece_id=WLBTOA5UHXE2WFXGJTJ2AYUZXWXY6ZYQ7MN4ZR44B4XRJ7PXV47A, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.048093444 (60h21m47.770656484s) [noise=true , address=77.90.31.10:4544 , piece_id=TIZSESFHKJXQXE2P2ATUSPYJHFP5VD7FM6D4YF6Z3UEUGEALGAQQ, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.048329967 (60h21m47.770893007s) [noise=true , address=104.36.82.56:4262 , piece_id=ULQXGOENYL6GJKQHHQUXZMXHZBZGBPLE7NGM7CF64ZELWJF3QGMA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049040106 (60h21m47.771603136s) [noise=true , address=45.91.116.14:4194 , piece_id=DD2FUOO3O6YAYJZ23TYGROGF3BEJDCOX7Z6ZFTLR32V4DY23IY3Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049242322 (60h21m47.771805352s) [noise=true , address=77.90.31.16:4736 , piece_id=I2OIN67ZONMXFUFS4LFG6WJGH5S4E7VHW5DRANKCFCRJJ4EJZB2Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049432847 (60h21m47.771995887s) [noise=true , address=104.36.82.46:4501 , piece_id=YESAAOIXCYHVCWOZXKWO4NAIEMJGB5TA4A3GDPCB333C6KOVYR3Q, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:38.049618332 (60h21m47.772181362s) [noise=true , address=104.36.82.54:4174 , piece_id=2HDRVUUXQZ7MQCZQNLO2FXZUODMNRITQFE66T46YWSFNMXVC7YDA, seg_pos={0 0}, upload=57899]: Uploading piece... Jul 26 23:57:39.149687765 (60h21m48.872250805s) [upload=57899]: Next returned nil segment Jul 26 23:57:39.149731879 (60h21m48.87229492s) [upload=57899]: Waiting for error group managing segments... Jul 26 23:57:39.455484025 (60h21m49.178047055s) [noise=true , address=79.127.205.246:4253 , piece_id=NATM45ZYE6H4RHXA4KNXW6CK7KQLPPRD4A7A6KOCJJWCSGYNJEEQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.473580770 (60h21m49.19614381s) [noise=true , address=79.127.205.245:4240 , piece_id=IFFO3URWPDROY7WJE3ZNRHLRU67NNJAD2LLQHOHYCRLFAYMOK6YQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.950821293 (60h21m49.673384333s) [noise=true , address=77.90.31.11:4585 , piece_id=56X44FTMER6FK647XF73E5EFU6NZQASTZH4YSK2Z77MQOJ635CUQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.958601049 (60h21m49.681164079s) [noise=true , address=77.90.31.16:4736 , piece_id=I2OIN67ZONMXFUFS4LFG6WJGH5S4E7VHW5DRANKCFCRJJ4EJZB2Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:39.982565981 (60h21m49.705129011s) [noise=true , address=79.127.219.38:4006 , piece_id=H76UQJJ47D6F6AUDYYDVUJX75PLRWT5PUL75YI2W7MR7KXGGSBLA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:40.160505642 (60h21m49.883068682s) [noise=true , address=64.49.13.138:4483 , piece_id=2OXH6RJVAIYXZKRO2DC7AMEAEYK4WINBV52HAQUK7WBPXYKXKZRA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:40.998834908 (60h21m50.721397948s) [noise=true , address=79.127.220.98:4421 , piece_id=6IJ3UDSZ3OONAIM4HJOH7FZMWZJDNL3JIZW2FQVF7G3GCJO5372A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:41.022368595 (60h21m50.744931675s) [noise=true , address=77.90.31.13:4629 , piece_id=ETIN7EO7P7GQ6O2RVKU5PRQAAKL47GCYD6AFEGUPI4VXILLPA6XA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:42.490649298 (60h21m52.213212358s) [noise=true , address=156.146.43.225:4448 , piece_id=SRHUMHAXD3JSLXF4QLOSSNWO2DO757HDGHQ3K73HG6XDLPVTIEYA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:57:46.089774242 (60h21m55.812337272s) [noise=true , address=45.91.116.14:4194 , piece_id=DD2FUOO3O6YAYJZ23TYGROGF3BEJDCOX7Z6ZFTLR32V4DY23IY3Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: <nil> Jul 26 23:58:36.954450190 (60h22m46.67701322s) [noise=true , address=104.36.82.56:4262 , piece_id=ULQXGOENYL6GJKQHHQUXZMXHZBZGBPLE7NGM7CF64ZELWJF3QGMA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954517127 (60h22m46.677080167s) [noise=true , address=104.36.82.46:4501 , piece_id=YESAAOIXCYHVCWOZXKWO4NAIEMJGB5TA4A3GDPCB333C6KOVYR3Q, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 188.166.225.170:41388->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954561182 (60h22m46.677124212s) [noise=true , address=104.36.82.47:4519 , piece_id=M6PHGB34J74SAE2ZYI44NJ3V4HNMFWS75VTDTH3FVP7R6RJCGKVA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: rpc: dial tcp 104.36.82.47:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954587533 (60h22m46.677150573s) [noise=true , address=104.36.82.53:4153 , piece_id=WLBTOA5UHXE2WFXGJTJ2AYUZXWXY6ZYQ7MN4ZR44B4XRJ7PXV47A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: rpc: dial tcp 104.36.82.53:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954610947 (60h22m46.677173988s) [noise=true , address=104.36.82.44:4894 , piece_id=YHXB52SOW4OPCIYBA5PQDGIBH3KLW3E2KWXGHXGHPHIP4WI7XKEA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954652166 (60h22m46.677215206s) [noise=true , address=104.36.82.55:4197 , piece_id=WNB5MAAIUHWHWBBY5EJE56WHRN2HT4EZLOLCVLWORCMYHPVOQ26A, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954667125 (60h22m46.677230165s) [noise=true , address=77.90.31.10:4544 , piece_id=TIZSESFHKJXQXE2P2ATUSPYJHFP5VD7FM6D4YF6Z3UEUGEALGAQQ, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: rpc: dial tcp 77.90.31.10:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954685730 (60h22m46.67724877s) [noise=true , address=104.36.82.54:4174 , piece_id=2HDRVUUXQZ7MQCZQNLO2FXZUODMNRITQFE66T46YWSFNMXVC7YDA, seg_pos={0 0}, upload=57899]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: write tcp 188.166.225.170:56578->104.36.82.54:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:36.954709726 (60h22m46.677272766s) [seg_pos={0 0}, upload=57899]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:58:37.182397114 (60h22m46.904960155s) [upload=57899]: Done uploading segments... Jul 26 23:58:37.182419267 (60h22m46.904982307s) [upload=57899]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230029483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX] duration: 30.232359179s FAIL (2025-07-26 23:58:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:44896->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:57:52.262108051 (60h22m1.984671091s) [seg_pos={0 0}, upload=57894]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:57:52.500328175 (60h22m2.222891205s) [upload=57894]: Done uploading segments... Jul 26 23:57:52.500353694 (60h22m2.222916734s) [upload=57894]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:57:53.376098356 (60h22m3.098661386s) [upload=57901]: Starting upload Jul 26 23:57:53.376203638 (60h22m3.098766668s) [upload=57901]: Done starting upload Jul 26 23:57:53.376274283 (60h22m3.098837324s) [upload=57901]: Uploading segments... Jul 26 23:57:53.376384194 (60h22m3.098947234s) [upload=57901]: Got next segment. Inline: false Jul 26 23:57:53.376410224 (60h22m3.098973264s) [upload=57901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:53.376414512 (60h22m3.098977552s) [upload=57901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:53.611631963 (60h22m3.334195003s) [seg_pos={0 0}, upload=57901]: Begin upload segment... Jul 26 23:57:53.611675807 (60h22m3.334238847s) [seg_pos={0 0}, upload=57901]: Done begin upload segment. Jul 26 23:57:53.611695715 (60h22m3.334258755s) [noise=true , address=45.91.116.5:4298 , piece_id=AFQ2QK24RUHRS6VXLFXSUXM7GOIUN2F3KFI7WZI2SRWB6UDFNNUA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.611725352 (60h22m3.334288392s) [noise=true , address=77.90.31.11:4553 , piece_id=QQ5PZQ5LHQLMFAYTPP6GGTYOIAWJIEZUL2DWCQ3IOZMCIM5XIBHA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612030927 (60h22m3.334593967s) [noise=true , address=77.90.31.16:4734 , piece_id=QBDKXFMXE3FF45VHMACJOOR5DN5LX6UICYBBENGEUWZRWHKNAWDQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612404340 (60h22m3.33496737s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XVCIODDL4TKYT26M5IVC5JHYSUVNZSYYQOBTXZN3MSCHR6LC2Q, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612560429 (60h22m3.335123469s) [noise=true , address=77.90.31.19:4841 , piece_id=ZPG2ATD2KCI5HKALKT67JACMLLZFYZPR4JMTB57FI7YXBAKQSHBQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612663056 (60h22m3.335226096s) [noise=true , address=77.90.31.19:4843 , piece_id=XV73NM2LA25COLBIJTVMSGEJJLI6SZYXS4KORLY4DUC7O6AFLGOQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.612897173 (60h22m3.335460213s) [noise=true , address=104.36.82.7:4910 , piece_id=Z55Q4HDRMYWIMEDT7RQKYPY5T32GDW7OEYQ67UHDQKUF3TKRZTWQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.613188291 (60h22m3.335751331s) [noise=true , address=45.91.116.11:4118 , piece_id=AWVRI7A6EKV6XKOIYHHB5IUHHUX2HMTBG3B3Z3AQSMEXIOUER4RA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.613430855 (60h22m3.335993895s) [noise=true , address=45.91.116.5:4299 , piece_id=NOEHVVHQLTKKMOZFQN5AHGMVNWJR6J4W76SFQHWQH35ZH2XHUGJA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614210076 (60h22m3.336773116s) [noise=true , address=104.36.82.10:4078 , piece_id=UK33Z4KGYR257OJGV7YXQNVYGPE4CG4ZSBFSDVXTUW6OIT7YOPAA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614459212 (60h22m3.337022252s) [noise=true , address=45.91.116.9:4262 , piece_id=NBMSQ2GR47WBAOEIUZNIQE3I354YUEMNKC2CY6ZBDSPVM24Q2NHQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.614778233 (60h22m3.337341273s) [noise=true , address=45.91.116.4:4939 , piece_id=7UQBD3YEUBVUMVSJELXDXS5HBSK6YJLK4SJW3XNWNGSNZNHT7FYQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615069850 (60h22m3.33763289s) [noise=true , address=77.90.31.18:4807 , piece_id=MEFRRCJG6COPKBQR2APTO5TCEOXP5O5MEJGREO4WORVFP6ZNF2IA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615309780 (60h22m3.33787282s) [noise=true , address=77.90.31.17:4770 , piece_id=IEJPIZFXE434BDQQZWWA3CKV6FDBLIUG4OBSZBHO342JZRY7FUEA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615542556 (60h22m3.338105596s) [noise=true , address=77.90.31.14:4661 , piece_id=FBKFSS4D4JOASPR4XHY2MKHXHD2HKPBIQESV4SBXTY75MGKDL3KA, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:53.615802703 (60h22m3.338365743s) [noise=true , address=45.91.116.8:4047 , piece_id=6A4WA7GCGF624XMCXVFEN7IT3TCQNZ7HT5VVTTB55AHR2VU4CLJQ, seg_pos={0 0}, upload=57901]: Uploading piece... Jul 26 23:57:55.324701039 (60h22m5.047264109s) [upload=57901]: Next returned nil segment Jul 26 23:57:55.324723832 (60h22m5.047286863s) [upload=57901]: Waiting for error group managing segments... Jul 26 23:57:55.622628072 (60h22m5.345191102s) [noise=true , address=45.91.116.5:4298 , piece_id=AFQ2QK24RUHRS6VXLFXSUXM7GOIUN2F3KFI7WZI2SRWB6UDFNNUA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.641771479 (60h22m5.364334519s) [noise=true , address=45.91.116.4:4939 , piece_id=7UQBD3YEUBVUMVSJELXDXS5HBSK6YJLK4SJW3XNWNGSNZNHT7FYQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.698429273 (60h22m5.420992313s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XVCIODDL4TKYT26M5IVC5JHYSUVNZSYYQOBTXZN3MSCHR6LC2Q, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.726265291 (60h22m5.448828331s) [noise=true , address=45.91.116.5:4299 , piece_id=NOEHVVHQLTKKMOZFQN5AHGMVNWJR6J4W76SFQHWQH35ZH2XHUGJA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.740361332 (60h22m5.462924372s) [noise=true , address=77.90.31.14:4661 , piece_id=FBKFSS4D4JOASPR4XHY2MKHXHD2HKPBIQESV4SBXTY75MGKDL3KA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.767343256 (60h22m5.489906296s) [noise=true , address=45.91.116.11:4118 , piece_id=AWVRI7A6EKV6XKOIYHHB5IUHHUX2HMTBG3B3Z3AQSMEXIOUER4RA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.798300466 (60h22m5.520863506s) [noise=true , address=77.90.31.11:4553 , piece_id=QQ5PZQ5LHQLMFAYTPP6GGTYOIAWJIEZUL2DWCQ3IOZMCIM5XIBHA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.801780304 (60h22m5.524343344s) [noise=true , address=45.91.116.8:4047 , piece_id=6A4WA7GCGF624XMCXVFEN7IT3TCQNZ7HT5VVTTB55AHR2VU4CLJQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.803751445 (60h22m5.526314485s) [noise=true , address=77.90.31.16:4734 , piece_id=QBDKXFMXE3FF45VHMACJOOR5DN5LX6UICYBBENGEUWZRWHKNAWDQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:55.807686584 (60h22m5.530249624s) [noise=true , address=45.91.116.9:4262 , piece_id=NBMSQ2GR47WBAOEIUZNIQE3I354YUEMNKC2CY6ZBDSPVM24Q2NHQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:57.091454785 (60h22m6.814017825s) [noise=true , address=77.90.31.19:4841 , piece_id=ZPG2ATD2KCI5HKALKT67JACMLLZFYZPR4JMTB57FI7YXBAKQSHBQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:57:57.562405322 (60h22m7.284968362s) [noise=true , address=77.90.31.19:4843 , piece_id=XV73NM2LA25COLBIJTVMSGEJJLI6SZYXS4KORLY4DUC7O6AFLGOQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: <nil> Jul 26 23:58:22.719662701 (60h22m32.442225731s) [noise=true , address=77.90.31.17:4770 , piece_id=IEJPIZFXE434BDQQZWWA3CKV6FDBLIUG4OBSZBHO342JZRY7FUEA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:46326->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719727925 (60h22m32.442290955s) [noise=true , address=104.36.82.10:4078 , piece_id=UK33Z4KGYR257OJGV7YXQNVYGPE4CG4ZSBFSDVXTUW6OIT7YOPAA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:34810->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719778071 (60h22m32.442341101s) [noise=true , address=77.90.31.18:4807 , piece_id=MEFRRCJG6COPKBQR2APTO5TCEOXP5O5MEJGREO4WORVFP6ZNF2IA, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35236->77.90.31.18:4807: use of closed network connection Jul 26 23:58:22.719793410 (60h22m32.44235645s) [noise=true , address=104.36.82.7:4910 , piece_id=Z55Q4HDRMYWIMEDT7RQKYPY5T32GDW7OEYQ67UHDQKUF3TKRZTWQ, seg_pos={0 0}, upload=57901]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:48550->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:22.719819740 (60h22m32.44238278s) [seg_pos={0 0}, upload=57901]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:58:22.950197325 (60h22m32.672760365s) [upload=57901]: Done uploading segments... Jul 26 23:58:22.950224778 (60h22m32.672787808s) [upload=57901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232373095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX..] duration: 13.380103738s FAIL (2025-07-26 23:57:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.380124128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.X] duration: 30.240920465s FAIL (2025-07-26 23:57:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e starting upload Jul 26 23:57:22.900804744 (60h21m32.623367784s) [upload=57894]: Uploading segments... Jul 26 23:57:22.900899145 (60h21m32.623462185s) [upload=57894]: Got next segment. Inline: false Jul 26 23:57:22.900941105 (60h21m32.623504145s) [upload=57894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:22.900945835 (60h21m32.623508865s) [upload=57894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:23.132328870 (60h21m32.85489191s) [seg_pos={0 0}, upload=57894]: Begin upload segment... Jul 26 23:57:23.132368847 (60h21m32.854931887s) [seg_pos={0 0}, upload=57894]: Done begin upload segment. Jul 26 23:57:23.132388755 (60h21m32.854951795s) [noise=true , address=104.36.82.8:4972 , piece_id=ESU44TKC6WEYX26477UNQY256ZXLKQ2DBAORGV6UOZEBNRDMB5AA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.132438159 (60h21m32.855001199s) [noise=true , address=77.90.31.19:4843 , piece_id=AQPCT6DM2C74MNM3GEH5VMJQQQYPE6EDOIFZLLLHVSLAKYEORFOQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.132881618 (60h21m32.855444658s) [noise=true , address=104.36.82.3:4765 , piece_id=FJBVENVGZFVVM7ZQTQZPNANFTEWR3IWKGOQ26KB3TEXQTV2MWDXQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133038177 (60h21m32.855601217s) [noise=true , address=77.90.31.18:4805 , piece_id=6T7UHMAGFCHGAZCMJNQXWKXSBLKWFRH4Y2CCLTEO5QMBFHNVQQDA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133175911 (60h21m32.855738951s) [noise=true , address=45.91.116.7:4010 , piece_id=UMX7Z456LBAXBQ5MM3VWX6QGNWBIOO73DPPMH4OC5C2KUF22JU7A, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133307493 (60h21m32.855870533s) [noise=true , address=104.36.82.9:4007 , piece_id=W2RSP2CTIHTQEECF72OAOTGKL4J6XHEJSKYDBQ7ZAJM62NNL6SSA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133455315 (60h21m32.856018355s) [noise=true , address=77.90.31.21:4878 , piece_id=JGQS42AAEIMXJLHLDXM7LT6M7WFQN3XDXGAPN2G52TWVAUFGWZFA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133733377 (60h21m32.856296417s) [noise=true , address=45.91.116.7:4009 , piece_id=K5NU7FUKXMMCPANCX6UGFDNVBLXLL2HZHYWV5AZEZXOA3CZEPGKQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.133781500 (60h21m32.85634454s) [noise=true , address=104.36.82.4:4802 , piece_id=N6IKNZJTWB3BPPYKB7JU4R6NBDHZ2VWQXGD7G2BMZKB4QW6U6JYA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134005458 (60h21m32.856568498s) [noise=true , address=45.91.116.11:4119 , piece_id=OR3O5ZPNKNAJDMBD5362SF4Y7WCOKTPP32W55CABPV73LVBW7BEA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134062096 (60h21m32.856625136s) [noise=true , address=45.91.116.12:4226 , piece_id=7UMQUC5FEMSOZQOAP3M752HC6MCNCA7ZGUMJD2PUJ2LCVBSW4DCQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134278200 (60h21m32.85684123s) [noise=true , address=77.90.31.11:4555 , piece_id=LNZ6J2JAVZDZUW2QDWNO3SJVY267ZQADYNPONOGSFSJC4IIMFIEA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134308598 (60h21m32.856871638s) [noise=true , address=45.91.116.10:4083 , piece_id=2TGO6QIKBTMOHBFO7TZWEPABGPVM2FGGB3HHLZA3AZ46T6T6WTLQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134523058 (60h21m32.857086099s) [noise=true , address=77.90.31.19:4841 , piece_id=AZMC5AMIK5WH3CKGSKVOT7QPCWRC55HS5HOPPUJNJUXKF2327AEQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134724854 (60h21m32.857287884s) [noise=true , address=104.36.82.7:4909 , piece_id=XUDABXU6CK3U7VALBAVEJ56OPE34Z2S2QPRDUCZBMAQVEIFWIBUQ, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:23.134922111 (60h21m32.857485152s) [noise=true , address=104.36.82.16:4295 , piece_id=7X2HWJUJO5M5QYBC7NBDY2RRTMGHD7BYPN3N6CGJUJ3EXGOLPYDA, seg_pos={0 0}, upload=57894]: Uploading piece... Jul 26 23:57:24.874325773 (60h21m34.596888803s) [upload=57894]: Next returned nil segment Jul 26 23:57:24.874342895 (60h21m34.596905935s) [upload=57894]: Waiting for error group managing segments... Jul 26 23:57:25.158541383 (60h21m34.881104423s) [noise=true , address=45.91.116.7:4009 , piece_id=K5NU7FUKXMMCPANCX6UGFDNVBLXLL2HZHYWV5AZEZXOA3CZEPGKQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.229145159 (60h21m34.951708199s) [noise=true , address=77.90.31.11:4555 , piece_id=LNZ6J2JAVZDZUW2QDWNO3SJVY267ZQADYNPONOGSFSJC4IIMFIEA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.235963203 (60h21m34.958526233s) [noise=true , address=45.91.116.12:4226 , piece_id=7UMQUC5FEMSOZQOAP3M752HC6MCNCA7ZGUMJD2PUJ2LCVBSW4DCQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.325026678 (60h21m35.047589718s) [noise=true , address=45.91.116.10:4083 , piece_id=2TGO6QIKBTMOHBFO7TZWEPABGPVM2FGGB3HHLZA3AZ46T6T6WTLQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.374807515 (60h21m35.097370555s) [noise=true , address=45.91.116.7:4010 , piece_id=UMX7Z456LBAXBQ5MM3VWX6QGNWBIOO73DPPMH4OC5C2KUF22JU7A, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:25.395860095 (60h21m35.118423135s) [noise=true , address=45.91.116.11:4119 , piece_id=OR3O5ZPNKNAJDMBD5362SF4Y7WCOKTPP32W55CABPV73LVBW7BEA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:26.596637682 (60h21m36.319200722s) [noise=true , address=77.90.31.19:4843 , piece_id=AQPCT6DM2C74MNM3GEH5VMJQQQYPE6EDOIFZLLLHVSLAKYEORFOQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:26.761740593 (60h21m36.484303633s) [noise=true , address=77.90.31.19:4841 , piece_id=AZMC5AMIK5WH3CKGSKVOT7QPCWRC55HS5HOPPUJNJUXKF2327AEQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: <nil> Jul 26 23:57:52.261458277 (60h22m1.984021317s) [noise=true , address=77.90.31.21:4878 , piece_id=JGQS42AAEIMXJLHLDXM7LT6M7WFQN3XDXGAPN2G52TWVAUFGWZFA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:52.261614286 (60h22m1.984177326s) [noise=true , address=104.36.82.7:4909 , piece_id=XUDABXU6CK3U7VALBAVEJ56OPE34Z2S2QPRDUCZBMAQVEIFWIBUQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:46122->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261863423 (60h22m1.984426463s) [noise=true , address=104.36.82.9:4007 , piece_id=W2RSP2CTIHTQEECF72OAOTGKL4J6XHEJSKYDBQ7ZAJM62NNL6SSA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:46614->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261895494 (60h22m1.984458534s) [noise=true , address=104.36.82.16:4295 , piece_id=7X2HWJUJO5M5QYBC7NBDY2RRTMGHD7BYPN3N6CGJUJ3EXGOLPYDA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:35576->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261926092 (60h22m1.984489123s) [noise=true , address=104.36.82.3:4765 , piece_id=FJBVENVGZFVVM7ZQTQZPNANFTEWR3IWKGOQ26KB3TEXQTV2MWDXQ, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:54596->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261957924 (60h22m1.984520964s) [noise=true , address=104.36.82.8:4972 , piece_id=ESU44TKC6WEYX26477UNQY256ZXLKQ2DBAORGV6UOZEBNRDMB5AA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:36806->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.261993041 (60h22m1.984556081s) [noise=true , address=104.36.82.4:4802 , piece_id=N6IKNZJTWB3BPPYKB7JU4R6NBDHZ2VWQXGD7G2BMZKB4QW6U6JYA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:50144->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:52.262093873 (60h22m1.984656903s) [noise=true , address=77.90.31.18:4805 , piece_id=6T7UHMAGFCHGAZCMJNQXWKXSBLKWFRH4Y2CCLTEO5QMBFHNVQQDA, seg_pos={0 0}, upload=57894]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:44896->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:57:52.262108051 (60h22m1.984671091s) [seg_pos={0 0}, upload=57894]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:57:52.500328175 (60h22m2.222891205s) [upload=57894]: Done uploading segments... Jul 26 23:57:52.500353694 (60h22m2.222916734s) [upload=57894]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.24094465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234016619s FAIL (2025-07-26 23:57:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== IPYN6UOOODGVLMPXGLPKXFY2A77XFI2K7XXMRC7ZA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.584936142 (60h20m47.307499182s) [noise=true , address=79.127.205.235:4686 , piece_id=UOQAFTB5QPNPGFKBVSLNHVMRDETAI2ZTBZEHXWOJIXR43YPRX5EQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.585335706 (60h20m47.307898746s) [noise=true , address=79.127.205.233:4633 , piece_id=NQQ7Z6OB4TXECBCIPYJ6I67H2PWXOVKXRX2ODQJIPLZMTJWJGZZA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.585460916 (60h20m47.308023956s) [noise=true , address=77.90.31.13:4644 , piece_id=YGAPCYWSQLGZJ6PVTYQLPP5LMAN4GXZXJF7443JSN4BKM5IJYWNA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.585645470 (60h20m47.3082085s) [noise=true , address=79.127.219.47:4048 , piece_id=5CIPGPC7LVSXYYJDAI6HCGBPO5IXPSVSMSMAAVCDIKTF4LSHZOZA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.585984869 (60h20m47.308547909s) [noise=true , address=104.36.82.55:4207 , piece_id=24W3TQ7GWMWHLWFBLK6W6XJSJEDIA5FWNM4FJGZAYLNIN5KMR3VA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.586358713 (60h20m47.308921753s) [noise=true , address=104.36.82.46:4492 , piece_id=J3ACMUKYTHOKRCOY3JEAC7F7ICFJKQGOBIL2XNGAWEKXXSZQMSTQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.586473945 (60h20m47.309036985s) [noise=true , address=104.36.82.15:4285 , piece_id=Q6DDDL5VAVUI6SSTTMJZN4CRVBIBCZCR4UFAAM55XESKUK5TLP2Q, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.586647256 (60h20m47.309210296s) [noise=true , address=104.36.82.56:4238 , piece_id=5XFFRLGIPMJJGLT37ILEDZWDEZABYFP723P5XUDJ5HBLKRP26JFQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.586896843 (60h20m47.309459873s) [noise=true , address=104.36.82.47:4552 , piece_id=BUGPVA3TUWAWM2KZUMXPAXGX3PH7QDTOTBSY2FPYFJCJXFK6VWRA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.586962819 (60h20m47.309525859s) [noise=true , address=45.91.116.14:4206 , piece_id=NNGWDTVHFRVYTM55WIOS4GDDT2BAFILWTKWWZKAAVTSNUBZNWGDA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.587444000 (60h20m47.31000704s) [noise=true , address=77.90.31.11:4585 , piece_id=QROPAI63UAK53Q5AIRSIN5KT3ZO4SK5HD2KXIFRWBVI2RXQG7EKA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.587897628 (60h20m47.310460658s) [noise=true , address=104.36.82.52:4089 , piece_id=UZJK6BBINHY5QKZU4GXXP6OZ6EWKZZ5DN4IY6Q544WYZUCDWF6AQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.588346678 (60h20m47.310909718s) [noise=true , address=79.127.219.38:4018 , piece_id=KHGOG4HR7XGUNKWALUIF5TXB7NCFGCND3ETHNSXLNE3UPIEPLLAA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.588404318 (60h20m47.310967358s) [noise=true , address=104.36.82.54:4184 , piece_id=KBGCWJIEZCNTZ4X64KQL72IFETMVYSYUZUJAYFEJAXTO6IV3YQAQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.588651511 (60h20m47.311214541s) [noise=true , address=104.36.82.50:4013 , piece_id=C5FOBA76RFICQIXC2DPR7IWH5NX2WSALCLA5UDUCJ5VOE33TF2ZQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.588867684 (60h20m47.311430724s) [noise=true , address=77.90.31.12:4608 , piece_id=B3AII5RD2XEAMYT4ECXF3WVUAVAYTT3A4764XXY26AGBJ5NL7Z3Q, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:37.589007001 (60h20m47.311570031s) [noise=true , address=79.127.205.246:4262 , piece_id=REFZGWW27CA6FADZCR7PX5ZSXTXJD27HIXRG5CR3YL37CAK67ABA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:56:38.678962964 (60h20m48.401526004s) [upload=57883]: Next returned nil segment Jul 26 23:56:38.678982632 (60h20m48.401545662s) [upload=57883]: Waiting for error group managing segments... Jul 26 23:56:38.971675902 (60h20m48.694238943s) [noise=true , address=79.127.205.235:4686 , piece_id=UOQAFTB5QPNPGFKBVSLNHVMRDETAI2ZTBZEHXWOJIXR43YPRX5EQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:39.220821883 (60h20m48.943384923s) [noise=true , address=79.127.205.233:4633 , piece_id=NQQ7Z6OB4TXECBCIPYJ6I67H2PWXOVKXRX2ODQJIPLZMTJWJGZZA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:39.280846168 (60h20m49.003409198s) [noise=true , address=79.127.205.246:4262 , piece_id=REFZGWW27CA6FADZCR7PX5ZSXTXJD27HIXRG5CR3YL37CAK67ABA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:39.477406326 (60h20m49.199969366s) [noise=true , address=77.90.31.11:4585 , piece_id=QROPAI63UAK53Q5AIRSIN5KT3ZO4SK5HD2KXIFRWBVI2RXQG7EKA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:39.555842167 (60h20m49.278405207s) [noise=true , address=79.127.219.38:4018 , piece_id=KHGOG4HR7XGUNKWALUIF5TXB7NCFGCND3ETHNSXLNE3UPIEPLLAA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:40.747205694 (60h20m50.469768734s) [noise=true , address=79.127.219.47:4048 , piece_id=5CIPGPC7LVSXYYJDAI6HCGBPO5IXPSVSMSMAAVCDIKTF4LSHZOZA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:42.988776984 (60h20m52.711340064s) [noise=true , address=77.90.31.12:4608 , piece_id=B3AII5RD2XEAMYT4ECXF3WVUAVAYTT3A4764XXY26AGBJ5NL7Z3Q, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:52.051925551 (60h21m1.774488591s) [noise=true , address=45.91.116.14:4206 , piece_id=NNGWDTVHFRVYTM55WIOS4GDDT2BAFILWTKWWZKAAVTSNUBZNWGDA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:56:54.679403546 (60h21m4.401966586s) [noise=true , address=77.90.31.13:4644 , piece_id=YGAPCYWSQLGZJ6PVTYQLPP5LMAN4GXZXJF7443JSN4BKM5IJYWNA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:57:36.498047963 (60h21m46.220611003s) [noise=true , address=104.36.82.50:4013 , piece_id=C5FOBA76RFICQIXC2DPR7IWH5NX2WSALCLA5UDUCJ5VOE33TF2ZQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:36.498081938 (60h21m46.220644978s) [noise=true , address=77.90.31.10:4536 , piece_id=JTGHEP5F5NHIPYN6UOOODGVLMPXGLPKXFY2A77XFI2K7XXMRC7ZA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: write tcp 188.166.225.170:54662->77.90.31.10:4536: use of closed network connection; write tcp 188.166.225.170:54662->77.90.31.10:4536: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54662->77.90.31.10:4536: use of closed network connection Jul 26 23:57:36.498149207 (60h21m46.220712237s) [noise=true , address=104.36.82.15:4285 , piece_id=Q6DDDL5VAVUI6SSTTMJZN4CRVBIBCZCR4UFAAM55XESKUK5TLP2Q, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: rpc: dial tcp 104.36.82.15:4285: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:36.498217838 (60h21m46.220780878s) [noise=true , address=104.36.82.56:4238 , piece_id=5XFFRLGIPMJJGLT37ILEDZWDEZABYFP723P5XUDJ5HBLKRP26JFQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:36.498257875 (60h21m46.220820915s) [noise=true , address=104.36.82.52:4089 , piece_id=UZJK6BBINHY5QKZU4GXXP6OZ6EWKZZ5DN4IY6Q544WYZUCDWF6AQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: rpc: dial tcp 104.36.82.52:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:36.498283955 (60h21m46.220846985s) [noise=true , address=104.36.82.55:4207 , piece_id=24W3TQ7GWMWHLWFBLK6W6XJSJEDIA5FWNM4FJGZAYLNIN5KMR3VA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:36.498347386 (60h21m46.220910416s) [noise=true , address=104.36.82.47:4552 , piece_id=BUGPVA3TUWAWM2KZUMXPAXGX3PH7QDTOTBSY2FPYFJCJXFK6VWRA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:36.498356874 (60h21m46.220919914s) [noise=true , address=104.36.82.46:4492 , piece_id=J3ACMUKYTHOKRCOY3JEAC7F7ICFJKQGOBIL2XNGAWEKXXSZQMSTQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:57:36.498411047 (60h21m46.220974087s) [noise=true , address=104.36.82.54:4184 , piece_id=KBGCWJIEZCNTZ4X64KQL72IFETMVYSYUZUJAYFEJAXTO6IV3YQAQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG, address:104.36.82.54:4184): protocol: rpc: dial tcp 104.36.82.54:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:36.498426718 (60h21m46.220989758s) [seg_pos={0 0}, upload=57883]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:57:36.729371694 (60h21m46.451934734s) [upload=57883]: Done uploading segments... Jul 26 23:57:36.729396271 (60h21m46.451959311s) [upload=57883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.234042869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 7.174069018s FAIL (2025-07-26 23:57:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.17408545s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.883533295s FAIL (2025-07-26 23:57:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.883561319s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX.] duration: 27.045448188s FAIL (2025-07-26 23:57:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.045462846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...] duration: 8.286663434s FAIL (2025-07-26 23:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.286685185s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [X..XXX............XXXX.....XXX..X.....X......X...XX..XX.X.XX] duration: 1m0.000722088s FAIL (2025-07-26 23:57:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000731987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.836225673s FAIL (2025-07-26 23:56:57 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.836252653s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 6.293516055s FAIL (2025-07-26 23:56:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.293540492s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.227290061s FAIL (2025-07-26 23:56:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.227315178s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.225870207s FAIL (2025-07-26 23:56:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 91.116.9:4273 , piece_id=ZUYTP2AVHORUJGVVSJ5RSPSVREVCXUJZU6Y4AHRC6P25YF5DS6FQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.351614778 (60h19m45.074177818s) [noise=true , address=104.36.82.46:4504 , piece_id=OLKFMFAGQFOICRNVIU5NCVZT57QRBNGVRMZCT4BFC2ZSGY2BAOIQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.351963135 (60h19m45.074526165s) [noise=true , address=104.36.82.47:4520 , piece_id=NIBTQWXTCQ4QHGGHFJLYGWYSF5AWA5TODEQMGMKYIUELUN7ASBKA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352280642 (60h19m45.074843682s) [noise=true , address=104.36.82.53:4122 , piece_id=LV2BPKVTKMQ2RWBFQ6JTJBZBRR2SO3D2DA5LYRRRAM4QCRUWPCOA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352324917 (60h19m45.074887957s) [noise=true , address=104.36.82.54:4183 , piece_id=IGW4DNRZ5Y7DH5UJD34FFLNPJKYK3AXS6PNE7L54NCC3TOKBACLA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352558443 (60h19m45.075121483s) [noise=true , address=77.90.31.12:4596 , piece_id=TNNEB7JMCUGWXOAF2PRFLSU6NKI64GYDOZMDOLAXFJKO655YQU2Q, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352611615 (60h19m45.075174645s) [noise=true , address=79.127.205.234:4603 , piece_id=BVJLAASIHETJCW7GXWUNW5IR3TEEJB2RFLVZBT56P3FMGFDWZAIA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352829953 (60h19m45.075392993s) [noise=true , address=45.91.116.10:4093 , piece_id=DHY74PZAUVTMFCDS4PIQM275NPXUZMUFV336A5OMT56O4K7KATJQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.352877063 (60h19m45.075440093s) [noise=true , address=45.91.116.5:4307 , piece_id=FWSGDONG7KZAXHOH2ATMUDHIKSK47YQ6Z76RQBBKFT5SGXFSAHAA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353090813 (60h19m45.075653843s) [noise=true , address=77.90.31.16:4766 , piece_id=I7OORENZDQ5S64CH5G62GA2O5UNVD6C6FQZKO5Q57I33ZWCBKMHQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353125429 (60h19m45.075688469s) [noise=true , address=104.36.82.14:4245 , piece_id=P7DTTKWCUMDAFJJKFC3OCGOFX6VQJRZOKMK6G4V3Z6BLZWRHZXDA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353297758 (60h19m45.075860798s) [noise=true , address=64.49.13.138:4483 , piece_id=XOK2WNAD7MAKNH35OITUODNWSAQRK5AKYJRBFWTDKK3KOOYI4BFA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353329418 (60h19m45.075892459s) [noise=true , address=79.127.205.245:4240 , piece_id=4EFREKQHFKW4SURH73M6R2UCH6ILDYD6AZHE4BLWZYQOFFOJ7MMA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353500146 (60h19m45.076063186s) [noise=true , address=79.127.205.235:4688 , piece_id=JUTIB6LRG7IFD6XVQIB2MXPVYNXWKJCK5BXXODGAW3ELQUMIB3DQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353555662 (60h19m45.076118702s) [noise=true , address=64.49.13.26:4953 , piece_id=HCGVWB4F35XA3ADRBXSBUZAPOJFCOZ54SKORJEMSYSSAUNR32HEA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353702222 (60h19m45.076265262s) [noise=true , address=77.90.31.17:4787 , piece_id=7R7VVOE7FKPRMDYPQRET4J7LSYBET7GBNTIBTT3IZFZLFFBWJK3A, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353824616 (60h19m45.076387656s) [noise=true , address=79.127.219.38:4015 , piece_id=4G6ZXVIOPOJKYSBXTPQ2F3RZKFX2EUKCBGFWFPPMGINYBFSTR7IA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:35.353929888 (60h19m45.076492928s) [noise=true , address=79.127.205.246:4260 , piece_id=LB72ZGPMDFZ7XV53O75WD2DFL6WQAD3ZV5I7CNF7H4XNBO2HQIOA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:55:36.448603595 (60h19m46.171166635s) [upload=57868]: Next returned nil segment Jul 26 23:55:36.448615708 (60h19m46.171178748s) [upload=57868]: Waiting for error group managing segments... Jul 26 23:55:36.738338204 (60h19m46.460901244s) [noise=true , address=79.127.205.234:4603 , piece_id=BVJLAASIHETJCW7GXWUNW5IR3TEEJB2RFLVZBT56P3FMGFDWZAIA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:36.762408588 (60h19m46.484971618s) [noise=true , address=79.127.205.235:4688 , piece_id=JUTIB6LRG7IFD6XVQIB2MXPVYNXWKJCK5BXXODGAW3ELQUMIB3DQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:36.779042003 (60h19m46.501605043s) [noise=true , address=79.127.205.246:4260 , piece_id=LB72ZGPMDFZ7XV53O75WD2DFL6WQAD3ZV5I7CNF7H4XNBO2HQIOA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:36.852676949 (60h19m46.575239989s) [noise=true , address=79.127.205.245:4240 , piece_id=4EFREKQHFKW4SURH73M6R2UCH6ILDYD6AZHE4BLWZYQOFFOJ7MMA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:37.174171054 (60h19m46.896734094s) [noise=true , address=45.91.116.10:4093 , piece_id=DHY74PZAUVTMFCDS4PIQM275NPXUZMUFV336A5OMT56O4K7KATJQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 26 23:55:37.272739825 (60h19m46.995302865s) [noise=true , address=77.90.31.16:4766 , piece_id=I7OORENZDQ5S64CH5G62GA2O5UNVD6C6FQZKO5Q57I33ZWCBKMHQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:37.379455484 (60h19m47.102018524s) [noise=true , address=79.127.219.38:4015 , piece_id=4G6ZXVIOPOJKYSBXTPQ2F3RZKFX2EUKCBGFWFPPMGINYBFSTR7IA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:37.722205648 (60h19m47.444768678s) [noise=true , address=45.91.116.5:4307 , piece_id=FWSGDONG7KZAXHOH2ATMUDHIKSK47YQ6Z76RQBBKFT5SGXFSAHAA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Jul 26 23:55:40.318801590 (60h19m50.04136464s) [noise=true , address=64.49.13.26:4953 , piece_id=HCGVWB4F35XA3ADRBXSBUZAPOJFCOZ54SKORJEMSYSSAUNR32HEA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:43.414348917 (60h19m53.136911957s) [noise=true , address=64.49.13.138:4483 , piece_id=XOK2WNAD7MAKNH35OITUODNWSAQRK5AKYJRBFWTDKK3KOOYI4BFA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:55:47.385056524 (60h19m57.107619564s) [noise=true , address=45.91.116.9:4273 , piece_id=ZUYTP2AVHORUJGVVSJ5RSPSVREVCXUJZU6Y4AHRC6P25YF5DS6FQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:56:34.277750219 (60h20m44.000313259s) [noise=true , address=77.90.31.12:4596 , piece_id=TNNEB7JMCUGWXOAF2PRFLSU6NKI64GYDOZMDOLAXFJKO655YQU2Q, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1xgzDjjKfA3YT4tjGq6Esw9xfhA4xWMWnDbyAmNFUcP85qyFGf, address:77.90.31.12:4596): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:56:34.278110098 (60h20m44.000673138s) [noise=true , address=77.90.31.17:4787 , piece_id=7R7VVOE7FKPRMDYPQRET4J7LSYBET7GBNTIBTT3IZFZLFFBWJK3A, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278155946 (60h20m44.000718986s) [noise=true , address=104.36.82.54:4183 , piece_id=IGW4DNRZ5Y7DH5UJD34FFLNPJKYK3AXS6PNE7L54NCC3TOKBACLA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278193487 (60h20m44.000756527s) [noise=true , address=104.36.82.14:4245 , piece_id=P7DTTKWCUMDAFJJKFC3OCGOFX6VQJRZOKMK6G4V3Z6BLZWRHZXDA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278201403 (60h20m44.000764443s) [noise=true , address=104.36.82.53:4122 , piece_id=LV2BPKVTKMQ2RWBFQ6JTJBZBRR2SO3D2DA5LYRRRAM4QCRUWPCOA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278238654 (60h20m44.000801694s) [noise=true , address=104.36.82.46:4504 , piece_id=OLKFMFAGQFOICRNVIU5NCVZT57QRBNGVRMZCT4BFC2ZSGY2BAOIQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:43912->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278248703 (60h20m44.000811733s) [noise=true , address=104.36.82.47:4520 , piece_id=NIBTQWXTCQ4QHGGHFJLYGWYSF5AWA5TODEQMGMKYIUELUN7ASBKA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: rpc: dial tcp 104.36.82.47:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:34.278275204 (60h20m44.000838244s) [seg_pos={0 0}, upload=57868]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:56:34.502614754 (60h20m44.225177794s) [upload=57868]: Done uploading segments... Jul 26 23:56:34.502638329 (60h20m44.225201359s) [upload=57868]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.225882059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X..] duration: 6.020385906s FAIL (2025-07-26 23:56:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.020427485s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.] duration: 6.992069735s FAIL (2025-07-26 23:56:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== p2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:41124->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:55:41.293031371 (60h19m51.015594411s) [seg_pos={0 0}, upload=57863]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:55:41.525405932 (60h19m51.247968972s) [upload=57863]: Done uploading segments... Jul 26 23:55:41.525430038 (60h19m51.247993068s) [upload=57863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:55:42.542339001 (60h19m52.264902041s) [upload=57870]: Starting upload Jul 26 23:55:42.542418752 (60h19m52.264981792s) [upload=57870]: Done starting upload Jul 26 23:55:42.542529524 (60h19m52.265092564s) [upload=57870]: Uploading segments... Jul 26 23:55:42.542639926 (60h19m52.265202966s) [upload=57870]: Got next segment. Inline: false Jul 26 23:55:42.542669553 (60h19m52.265232593s) [upload=57870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:42.542674583 (60h19m52.265237623s) [upload=57870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:42.777916383 (60h19m52.500479423s) [seg_pos={0 0}, upload=57870]: Begin upload segment... Jul 26 23:55:42.777963733 (60h19m52.500526763s) [seg_pos={0 0}, upload=57870]: Done begin upload segment. Jul 26 23:55:42.777986176 (60h19m52.500549216s) [noise=true , address=77.90.31.21:4878 , piece_id=FIPOL7J3BBGCDU73YZAPUR4RKDIGGVMA6L27VP5FIZ52LUFFUSBQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778024409 (60h19m52.500587449s) [noise=true , address=45.91.116.10:4081 , piece_id=PZ2A7UTCO4XKX6VFZ77QRFASONJF7AXVJZCRXOXOQTTF33GGGVFA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778430947 (60h19m52.500993987s) [noise=true , address=45.91.116.5:4299 , piece_id=WSS7HXCMDQIHITOYD5Z6HKBI7BBDM5MGGJ6WLKCYJXFKU7J544LQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778513906 (60h19m52.501076976s) [noise=true , address=104.36.82.11:4113 , piece_id=3UWOOHTJ7CUYJZQYJ7EWTEYP3ABSPN5J3FFH6QXWQ7KRLCKNZFAA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778691456 (60h19m52.501254496s) [noise=true , address=104.36.82.16:4295 , piece_id=RYWXCSKFZIBGNX5ZPTK7LY3B3PBNXMELLNULSF6O4WETINLBV2BQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779208284 (60h19m52.501771324s) [noise=true , address=77.90.31.16:4735 , piece_id=DPJURVTGLECOEBVCPFU6WREY3J67VDQSYUHSMR4BQ3DYWSSKILIQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779484844 (60h19m52.502047884s) [noise=true , address=104.36.82.2:4731 , piece_id=VWXXXADTCLWM3WJXXS6PZSEWMYDFBZPYQLFIU6WUK4VYGOCM5GYA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779711888 (60h19m52.502274918s) [noise=true , address=104.36.82.3:4765 , piece_id=HJNUYK7RMH5OY5TRAAUIXWTNX4JP4KOSPIB6WQCET3ZL5O6KP4SQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779947018 (60h19m52.502510058s) [noise=true , address=45.91.116.8:4047 , piece_id=AE66E7VX2EOXMWSDA67LSMB24TIZSTBSL7QIMAUKEZBDO4CVC7LA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780231723 (60h19m52.502794763s) [noise=true , address=45.91.116.9:4263 , piece_id=SHMMN4O4C7AEYJULLRLZ2EW5TRTJCC357SKWIQZSGNP3UQ42B4OA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780473024 (60h19m52.503036054s) [noise=true , address=77.90.31.12:4589 , piece_id=GFFFCPVLGTLEDFKDRP2SF4DKBFJAARQW75RAJEI2C2B6SBSUG4BA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780678578 (60h19m52.503241618s) [noise=true , address=77.90.31.19:4841 , piece_id=AP4623SCJH6ZSZZ6VCYCZDG7DZ4W2HCUOE56W7K2BFU2KPB4S5PQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780864343 (60h19m52.503427383s) [noise=true , address=45.91.116.8:4045 , piece_id=37RA646JNGD7TLCLTCIEOSQ3OJMZKVG47MA2BLT5FXGAD35EHGAA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781087230 (60h19m52.50365027s) [noise=true , address=45.91.116.7:4010 , piece_id=VXZKSSNQ4U7DAW34SGREDCNKSWMY56QETZMKFHB4W3NZILB44KLQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781284988 (60h19m52.503848018s) [noise=true , address=77.90.31.14:4663 , piece_id=IE4FPGL7P7SSB4AD4C7B6EO4FPTQMP4STBSNKFZ64LMGHSWULUDQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781491634 (60h19m52.504054674s) [noise=true , address=45.91.116.4:4938 , piece_id=C5UTU2SYJJ64U3BEPDL7ZUZMXNYPD3QV4NMOZPKDFXNC6EDRGZMA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:44.537596718 (60h19m54.260159758s) [upload=57870]: Next returned nil segment Jul 26 23:55:44.537614822 (60h19m54.260177862s) [upload=57870]: Waiting for error group managing segments... Jul 26 23:55:44.873509860 (60h19m54.59607289s) [noise=true , address=45.91.116.4:4938 , piece_id=C5UTU2SYJJ64U3BEPDL7ZUZMXNYPD3QV4NMOZPKDFXNC6EDRGZMA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.886842430 (60h19m54.60940547s) [noise=true , address=45.91.116.8:4045 , piece_id=37RA646JNGD7TLCLTCIEOSQ3OJMZKVG47MA2BLT5FXGAD35EHGAA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.887125060 (60h19m54.6096881s) [noise=true , address=45.91.116.5:4299 , piece_id=WSS7HXCMDQIHITOYD5Z6HKBI7BBDM5MGGJ6WLKCYJXFKU7J544LQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.892944284 (60h19m54.615507324s) [noise=true , address=45.91.116.7:4010 , piece_id=VXZKSSNQ4U7DAW34SGREDCNKSWMY56QETZMKFHB4W3NZILB44KLQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.893327097 (60h19m54.615890137s) [noise=true , address=45.91.116.8:4047 , piece_id=AE66E7VX2EOXMWSDA67LSMB24TIZSTBSL7QIMAUKEZBDO4CVC7LA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.909288637 (60h19m54.631851677s) [noise=true , address=45.91.116.10:4081 , piece_id=PZ2A7UTCO4XKX6VFZ77QRFASONJF7AXVJZCRXOXOQTTF33GGGVFA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.944932534 (60h19m54.667495574s) [noise=true , address=77.90.31.16:4735 , piece_id=DPJURVTGLECOEBVCPFU6WREY3J67VDQSYUHSMR4BQ3DYWSSKILIQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.980273201 (60h19m54.702836241s) [noise=true , address=77.90.31.14:4663 , piece_id=IE4FPGL7P7SSB4AD4C7B6EO4FPTQMP4STBSNKFZ64LMGHSWULUDQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:45.438567346 (60h19m55.161130386s) [noise=true , address=45.91.116.9:4263 , piece_id=SHMMN4O4C7AEYJULLRLZ2EW5TRTJCC357SKWIQZSGNP3UQ42B4OA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:46.085480906 (60h19m55.808043936s) [noise=true , address=77.90.31.19:4841 , piece_id=AP4623SCJH6ZSZZ6VCYCZDG7DZ4W2HCUOE56W7K2BFU2KPB4S5PQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:46.556881959 (60h19m56.279445039s) [noise=true , address=77.90.31.12:4589 , piece_id=GFFFCPVLGTLEDFKDRP2SF4DKBFJAARQW75RAJEI2C2B6SBSUG4BA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:56:11.887795728 (60h20m21.610358768s) [noise=true , address=104.36.82.2:4731 , piece_id=VWXXXADTCLWM3WJXXS6PZSEWMYDFBZPYQLFIU6WUK4VYGOCM5GYA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36310->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887839081 (60h20m21.610402121s) [noise=true , address=104.36.82.3:4765 , piece_id=HJNUYK7RMH5OY5TRAAUIXWTNX4JP4KOSPIB6WQCET3ZL5O6KP4SQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:49516->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887859700 (60h20m21.61042274s) [noise=true , address=104.36.82.11:4113 , piece_id=3UWOOHTJ7CUYJZQYJ7EWTEYP3ABSPN5J3FFH6QXWQ7KRLCKNZFAA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887875450 (60h20m21.61043849s) [noise=true , address=77.90.31.21:4878 , piece_id=FIPOL7J3BBGCDU73YZAPUR4RKDIGGVMA6L27VP5FIZ52LUFFUSBQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:56:11.887899015 (60h20m21.610462055s) [noise=true , address=104.36.82.16:4295 , piece_id=RYWXCSKFZIBGNX5ZPTK7LY3B3PBNXMELLNULSF6O4WETINLBV2BQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:52800->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887918794 (60h20m21.610481834s) [seg_pos={0 0}, upload=57870]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:56:12.130907193 (60h20m21.853470233s) [upload=57870]: Done uploading segments... Jul 26 23:56:12.130932522 (60h20m21.853495562s) [upload=57870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.992085395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 30.245126304s FAIL (2025-07-26 23:56:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== p2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:41124->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:55:41.293031371 (60h19m51.015594411s) [seg_pos={0 0}, upload=57863]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:55:41.525405932 (60h19m51.247968972s) [upload=57863]: Done uploading segments... Jul 26 23:55:41.525430038 (60h19m51.247993068s) [upload=57863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:55:42.542339001 (60h19m52.264902041s) [upload=57870]: Starting upload Jul 26 23:55:42.542418752 (60h19m52.264981792s) [upload=57870]: Done starting upload Jul 26 23:55:42.542529524 (60h19m52.265092564s) [upload=57870]: Uploading segments... Jul 26 23:55:42.542639926 (60h19m52.265202966s) [upload=57870]: Got next segment. Inline: false Jul 26 23:55:42.542669553 (60h19m52.265232593s) [upload=57870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:42.542674583 (60h19m52.265237623s) [upload=57870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:42.777916383 (60h19m52.500479423s) [seg_pos={0 0}, upload=57870]: Begin upload segment... Jul 26 23:55:42.777963733 (60h19m52.500526763s) [seg_pos={0 0}, upload=57870]: Done begin upload segment. Jul 26 23:55:42.777986176 (60h19m52.500549216s) [noise=true , address=77.90.31.21:4878 , piece_id=FIPOL7J3BBGCDU73YZAPUR4RKDIGGVMA6L27VP5FIZ52LUFFUSBQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778024409 (60h19m52.500587449s) [noise=true , address=45.91.116.10:4081 , piece_id=PZ2A7UTCO4XKX6VFZ77QRFASONJF7AXVJZCRXOXOQTTF33GGGVFA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778430947 (60h19m52.500993987s) [noise=true , address=45.91.116.5:4299 , piece_id=WSS7HXCMDQIHITOYD5Z6HKBI7BBDM5MGGJ6WLKCYJXFKU7J544LQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778513906 (60h19m52.501076976s) [noise=true , address=104.36.82.11:4113 , piece_id=3UWOOHTJ7CUYJZQYJ7EWTEYP3ABSPN5J3FFH6QXWQ7KRLCKNZFAA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.778691456 (60h19m52.501254496s) [noise=true , address=104.36.82.16:4295 , piece_id=RYWXCSKFZIBGNX5ZPTK7LY3B3PBNXMELLNULSF6O4WETINLBV2BQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779208284 (60h19m52.501771324s) [noise=true , address=77.90.31.16:4735 , piece_id=DPJURVTGLECOEBVCPFU6WREY3J67VDQSYUHSMR4BQ3DYWSSKILIQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779484844 (60h19m52.502047884s) [noise=true , address=104.36.82.2:4731 , piece_id=VWXXXADTCLWM3WJXXS6PZSEWMYDFBZPYQLFIU6WUK4VYGOCM5GYA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779711888 (60h19m52.502274918s) [noise=true , address=104.36.82.3:4765 , piece_id=HJNUYK7RMH5OY5TRAAUIXWTNX4JP4KOSPIB6WQCET3ZL5O6KP4SQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.779947018 (60h19m52.502510058s) [noise=true , address=45.91.116.8:4047 , piece_id=AE66E7VX2EOXMWSDA67LSMB24TIZSTBSL7QIMAUKEZBDO4CVC7LA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780231723 (60h19m52.502794763s) [noise=true , address=45.91.116.9:4263 , piece_id=SHMMN4O4C7AEYJULLRLZ2EW5TRTJCC357SKWIQZSGNP3UQ42B4OA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780473024 (60h19m52.503036054s) [noise=true , address=77.90.31.12:4589 , piece_id=GFFFCPVLGTLEDFKDRP2SF4DKBFJAARQW75RAJEI2C2B6SBSUG4BA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780678578 (60h19m52.503241618s) [noise=true , address=77.90.31.19:4841 , piece_id=AP4623SCJH6ZSZZ6VCYCZDG7DZ4W2HCUOE56W7K2BFU2KPB4S5PQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.780864343 (60h19m52.503427383s) [noise=true , address=45.91.116.8:4045 , piece_id=37RA646JNGD7TLCLTCIEOSQ3OJMZKVG47MA2BLT5FXGAD35EHGAA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781087230 (60h19m52.50365027s) [noise=true , address=45.91.116.7:4010 , piece_id=VXZKSSNQ4U7DAW34SGREDCNKSWMY56QETZMKFHB4W3NZILB44KLQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781284988 (60h19m52.503848018s) [noise=true , address=77.90.31.14:4663 , piece_id=IE4FPGL7P7SSB4AD4C7B6EO4FPTQMP4STBSNKFZ64LMGHSWULUDQ, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:42.781491634 (60h19m52.504054674s) [noise=true , address=45.91.116.4:4938 , piece_id=C5UTU2SYJJ64U3BEPDL7ZUZMXNYPD3QV4NMOZPKDFXNC6EDRGZMA, seg_pos={0 0}, upload=57870]: Uploading piece... Jul 26 23:55:44.537596718 (60h19m54.260159758s) [upload=57870]: Next returned nil segment Jul 26 23:55:44.537614822 (60h19m54.260177862s) [upload=57870]: Waiting for error group managing segments... Jul 26 23:55:44.873509860 (60h19m54.59607289s) [noise=true , address=45.91.116.4:4938 , piece_id=C5UTU2SYJJ64U3BEPDL7ZUZMXNYPD3QV4NMOZPKDFXNC6EDRGZMA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.886842430 (60h19m54.60940547s) [noise=true , address=45.91.116.8:4045 , piece_id=37RA646JNGD7TLCLTCIEOSQ3OJMZKVG47MA2BLT5FXGAD35EHGAA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.887125060 (60h19m54.6096881s) [noise=true , address=45.91.116.5:4299 , piece_id=WSS7HXCMDQIHITOYD5Z6HKBI7BBDM5MGGJ6WLKCYJXFKU7J544LQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.892944284 (60h19m54.615507324s) [noise=true , address=45.91.116.7:4010 , piece_id=VXZKSSNQ4U7DAW34SGREDCNKSWMY56QETZMKFHB4W3NZILB44KLQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.893327097 (60h19m54.615890137s) [noise=true , address=45.91.116.8:4047 , piece_id=AE66E7VX2EOXMWSDA67LSMB24TIZSTBSL7QIMAUKEZBDO4CVC7LA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.909288637 (60h19m54.631851677s) [noise=true , address=45.91.116.10:4081 , piece_id=PZ2A7UTCO4XKX6VFZ77QRFASONJF7AXVJZCRXOXOQTTF33GGGVFA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.944932534 (60h19m54.667495574s) [noise=true , address=77.90.31.16:4735 , piece_id=DPJURVTGLECOEBVCPFU6WREY3J67VDQSYUHSMR4BQ3DYWSSKILIQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:44.980273201 (60h19m54.702836241s) [noise=true , address=77.90.31.14:4663 , piece_id=IE4FPGL7P7SSB4AD4C7B6EO4FPTQMP4STBSNKFZ64LMGHSWULUDQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:45.438567346 (60h19m55.161130386s) [noise=true , address=45.91.116.9:4263 , piece_id=SHMMN4O4C7AEYJULLRLZ2EW5TRTJCC357SKWIQZSGNP3UQ42B4OA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:46.085480906 (60h19m55.808043936s) [noise=true , address=77.90.31.19:4841 , piece_id=AP4623SCJH6ZSZZ6VCYCZDG7DZ4W2HCUOE56W7K2BFU2KPB4S5PQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:55:46.556881959 (60h19m56.279445039s) [noise=true , address=77.90.31.12:4589 , piece_id=GFFFCPVLGTLEDFKDRP2SF4DKBFJAARQW75RAJEI2C2B6SBSUG4BA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: <nil> Jul 26 23:56:11.887795728 (60h20m21.610358768s) [noise=true , address=104.36.82.2:4731 , piece_id=VWXXXADTCLWM3WJXXS6PZSEWMYDFBZPYQLFIU6WUK4VYGOCM5GYA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36310->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887839081 (60h20m21.610402121s) [noise=true , address=104.36.82.3:4765 , piece_id=HJNUYK7RMH5OY5TRAAUIXWTNX4JP4KOSPIB6WQCET3ZL5O6KP4SQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:49516->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887859700 (60h20m21.61042274s) [noise=true , address=104.36.82.11:4113 , piece_id=3UWOOHTJ7CUYJZQYJ7EWTEYP3ABSPN5J3FFH6QXWQ7KRLCKNZFAA, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887875450 (60h20m21.61043849s) [noise=true , address=77.90.31.21:4878 , piece_id=FIPOL7J3BBGCDU73YZAPUR4RKDIGGVMA6L27VP5FIZ52LUFFUSBQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:56:11.887899015 (60h20m21.610462055s) [noise=true , address=104.36.82.16:4295 , piece_id=RYWXCSKFZIBGNX5ZPTK7LY3B3PBNXMELLNULSF6O4WETINLBV2BQ, seg_pos={0 0}, upload=57870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:52800->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:11.887918794 (60h20m21.610481834s) [seg_pos={0 0}, upload=57870]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:56:12.130907193 (60h20m21.853470233s) [upload=57870]: Done uploading segments... Jul 26 23:56:12.130932522 (60h20m21.853495562s) [upload=57870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.245148047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XX..XXX............XXXX.....XXX..X.....X......X...XX..XX.X.X] duration: 1m0.000795847s FAIL (2025-07-26 23:56:01 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000820674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.865438442s FAIL (2025-07-26 23:55:56 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.865542541s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 30.234852831s FAIL (2025-07-26 23:55:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 04.36.82.15:4258 , piece_id=J3GOAH74ZLCTDFSVU3RAKTNRF3DSZLJAWXV25OGGLZS7K745Q2YQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:32866->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839766440 (60h19m20.56232948s) [seg_pos={0 0}, upload=57856]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:55:11.071214981 (60h19m20.793778041s) [upload=57856]: Done uploading segments... Jul 26 23:55:11.071241472 (60h19m20.793804522s) [upload=57856]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:55:11.938353518 (60h19m21.660916568s) [upload=57863]: Starting upload Jul 26 23:55:11.938456885 (60h19m21.661019925s) [upload=57863]: Done starting upload Jul 26 23:55:11.938641548 (60h19m21.661204589s) [upload=57863]: Uploading segments... Jul 26 23:55:11.938674030 (60h19m21.66123707s) [upload=57863]: Got next segment. Inline: false Jul 26 23:55:11.938698527 (60h19m21.661261567s) [upload=57863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:11.938703457 (60h19m21.661266497s) [upload=57863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:12.177303240 (60h19m21.89986634s) [seg_pos={0 0}, upload=57863]: Begin upload segment... Jul 26 23:55:12.177371501 (60h19m21.899934561s) [seg_pos={0 0}, upload=57863]: Done begin upload segment. Jul 26 23:55:12.177421256 (60h19m21.899984296s) [noise=true , address=45.91.116.6:4974 , piece_id=TXN5XP5XCGWQRGE7UP7PBCKCV5FQC7O6662QILOL6IIXLGKF7B2A, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.177428320 (60h19m21.89999136s) [noise=true , address=104.36.82.9:4006 , piece_id=ZPXTM2QLPZE7S7CM3SRWOLLXVNSRZ73VVDODOIKFIFFJZH3GZ5PA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.177890754 (60h19m21.900453794s) [noise=true , address=45.91.116.14:4189 , piece_id=25YI352XRNDVRX7HY2OEEA56SWK7AAC3G2OHLE6QSZYTJQC4VYCQ, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.177901836 (60h19m21.900464866s) [noise=true , address=77.90.31.21:4877 , piece_id=KQDAQEZWRKHYH5JHVBEUEXMN75GFF32APPGJF7OGCC4VI4OLQ2EA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.178254751 (60h19m21.900817821s) [noise=true , address=45.91.116.9:4263 , piece_id=CIEMGCV7B2FUM6A27QHFTFLTRAEHAI3PE24CMOEVGF4NS5ZUMXAA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.178566517 (60h19m21.901129567s) [noise=true , address=77.90.31.18:4807 , piece_id=MOETFGZM245GFPPAIAZY73IB3FFD3NNFSID4YMQN354HFIKAXQIA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.178805875 (60h19m21.901368915s) [noise=true , address=45.91.116.5:4298 , piece_id=YXABUMMEOYLEPXW6SBPXP5TMTZDUNSODK6R75LFOJQEBOAEYUPPQ, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.179065832 (60h19m21.901628872s) [noise=true , address=45.91.116.10:4083 , piece_id=LDTKVHEQBIQQXDB2TMN36Q2KUTKI4G7CTNLM3IPCOHU5JGO2YRMA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.179350468 (60h19m21.901913508s) [noise=true , address=77.90.31.13:4626 , piece_id=27V7HCK4MOLORHCKTRZFCHVXG2GX3FFFSJW7A2ECAFXLSVOY3WSQ, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.179632777 (60h19m21.902195818s) [noise=true , address=45.91.116.8:4048 , piece_id=7IWGJOVENH3GPFM5AGDP2GR47ZGASDG7UYFSM24JRUKTEMYOAR7A, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.179928884 (60h19m21.902491914s) [noise=true , address=77.90.31.14:4662 , piece_id=W37INHZC246ITNNX3LCQ5Q5AHICPSVIEBZNZLKBBYRSKLVTGNBQQ, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.180239178 (60h19m21.902802218s) [noise=true , address=45.91.116.11:4119 , piece_id=EL43VDWATFQKXSEKRJPPIUNZMY7STUC3MS53GP7YJNOBOK26W6KA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.180541686 (60h19m21.903104726s) [noise=true , address=45.91.116.4:4937 , piece_id=M7FEN2W35O6BM5TF5VD4XOIBEH3GN5UMMY5OJ46BUSWWUWEDWIEA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.180756387 (60h19m21.903319427s) [noise=true , address=45.91.116.13:4154 , piece_id=ZRB4FZAYVHKOJVO2KIFVRM5FZ3754QT6H5LNL4AIDB2A3AHGFIWQ, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.181462428 (60h19m21.904025458s) [noise=true , address=45.91.116.11:4118 , piece_id=R6Y43TGUQKVQ67WY56EELDSXOLTRFUHFQ4XJOYZAXMB3XWHHKO4Q, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:12.181766710 (60h19m21.90432975s) [noise=true , address=104.36.82.15:4257 , piece_id=V7ZXXCKRE4G63HDF44D47GDP6FVB6XY4NVAS5HONFHA5KDSYUQOA, seg_pos={0 0}, upload=57863]: Uploading piece... Jul 26 23:55:13.923065067 (60h19m23.645628097s) [upload=57863]: Next returned nil segment Jul 26 23:55:13.923081518 (60h19m23.645644558s) [upload=57863]: Waiting for error group managing segments... Jul 26 23:55:14.210657708 (60h19m23.933220748s) [noise=true , address=45.91.116.10:4083 , piece_id=LDTKVHEQBIQQXDB2TMN36Q2KUTKI4G7CTNLM3IPCOHU5JGO2YRMA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.212657554 (60h19m23.935220584s) [noise=true , address=45.91.116.5:4298 , piece_id=YXABUMMEOYLEPXW6SBPXP5TMTZDUNSODK6R75LFOJQEBOAEYUPPQ, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.294101771 (60h19m24.016664811s) [noise=true , address=45.91.116.6:4974 , piece_id=TXN5XP5XCGWQRGE7UP7PBCKCV5FQC7O6662QILOL6IIXLGKF7B2A, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.296348489 (60h19m24.01891152s) [noise=true , address=77.90.31.14:4662 , piece_id=W37INHZC246ITNNX3LCQ5Q5AHICPSVIEBZNZLKBBYRSKLVTGNBQQ, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.296828348 (60h19m24.019391418s) [noise=true , address=45.91.116.13:4154 , piece_id=ZRB4FZAYVHKOJVO2KIFVRM5FZ3754QT6H5LNL4AIDB2A3AHGFIWQ, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.297679286 (60h19m24.020242356s) [noise=true , address=45.91.116.8:4048 , piece_id=7IWGJOVENH3GPFM5AGDP2GR47ZGASDG7UYFSM24JRUKTEMYOAR7A, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.301025439 (60h19m24.023588479s) [noise=true , address=45.91.116.14:4189 , piece_id=25YI352XRNDVRX7HY2OEEA56SWK7AAC3G2OHLE6QSZYTJQC4VYCQ, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.306055773 (60h19m24.028618813s) [noise=true , address=45.91.116.11:4119 , piece_id=EL43VDWATFQKXSEKRJPPIUNZMY7STUC3MS53GP7YJNOBOK26W6KA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.363539336 (60h19m24.086102376s) [noise=true , address=45.91.116.4:4937 , piece_id=M7FEN2W35O6BM5TF5VD4XOIBEH3GN5UMMY5OJ46BUSWWUWEDWIEA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.456355141 (60h19m24.178918181s) [noise=true , address=45.91.116.11:4118 , piece_id=R6Y43TGUQKVQ67WY56EELDSXOLTRFUHFQ4XJOYZAXMB3XWHHKO4Q, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:14.650347457 (60h19m24.372910497s) [noise=true , address=45.91.116.9:4263 , piece_id=CIEMGCV7B2FUM6A27QHFTFLTRAEHAI3PE24CMOEVGF4NS5ZUMXAA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:16.429607706 (60h19m26.152170746s) [noise=true , address=77.90.31.13:4626 , piece_id=27V7HCK4MOLORHCKTRZFCHVXG2GX3FFFSJW7A2ECAFXLSVOY3WSQ, seg_pos={0 0}, upload=57863]: Done uploading piece. err: <nil> Jul 26 23:55:41.292877106 (60h19m51.015440146s) [noise=true , address=77.90.31.18:4807 , piece_id=MOETFGZM245GFPPAIAZY73IB3FFD3NNFSID4YMQN354HFIKAXQIA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:56914->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:41.292905700 (60h19m51.01546874s) [noise=true , address=77.90.31.21:4877 , piece_id=KQDAQEZWRKHYH5JHVBEUEXMN75GFF32APPGJF7OGCC4VI4OLQ2EA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:38290->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:41.292929947 (60h19m51.015492997s) [noise=true , address=104.36.82.15:4257 , piece_id=V7ZXXCKRE4G63HDF44D47GDP6FVB6XY4NVAS5HONFHA5KDSYUQOA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:49844->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:41.292999701 (60h19m51.015562751s) [noise=true , address=104.36.82.9:4006 , piece_id=ZPXTM2QLPZE7S7CM3SRWOLLXVNSRZ73VVDODOIKFIFFJZH3GZ5PA, seg_pos={0 0}, upload=57863]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:41124->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:55:41.293031371 (60h19m51.015594411s) [seg_pos={0 0}, upload=57863]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:55:41.525405932 (60h19m51.247968972s) [upload=57863]: Done uploading segments... Jul 26 23:55:41.525430038 (60h19m51.247993068s) [upload=57863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.234868261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235663896s FAIL (2025-07-26 23:55:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 6633s) [upload=57853]: Got next segment. Inline: false Jul 26 23:54:34.443567227 (60h18m44.166130267s) [upload=57853]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:34.443571866 (60h18m44.166134896s) [upload=57853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:34.963491797 (60h18m44.686054837s) [seg_pos={0 0}, upload=57853]: Begin upload segment... Jul 26 23:54:34.963553374 (60h18m44.686116415s) [seg_pos={0 0}, upload=57853]: Done begin upload segment. Jul 26 23:54:34.963563855 (60h18m44.686126895s) [noise=true , address=64.49.13.138:4482 , piece_id=666D7V6XKZIKTAXCEPV76IZ5UH6QIQM3QTQIIBXURD2ESFNWI6CA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.963575557 (60h18m44.686138597s) [noise=true , address=156.146.43.225:4449 , piece_id=3NNZDMPEHEG25BKQYLLFR66K57AZDBCVXFO3OXJGHX3UNFYVI6GA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.963929264 (60h18m44.686492304s) [noise=true , address=77.90.31.12:4596 , piece_id=D2ZHHBHBK6KEMKT5BTXOCPGWFXF3BEPKWRGNRO5ANLT6LKKBADHA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.964222776 (60h18m44.686785816s) [noise=true , address=79.127.205.233:4647 , piece_id=OSFVPA6XPHZQHT74247OHBFSBSNEKMHESV4FOJC5ZXLJWBYYOA6Q, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.964488233 (60h18m44.687051273s) [noise=true , address=77.90.31.10:4542 , piece_id=7HLQKYCD2KSJDGKQVRLEZYAJ4PVD4FLCUHJJ4MBL3N2S7TRAFPRQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.964769933 (60h18m44.687332973s) [noise=true , address=156.146.43.225:4448 , piece_id=K77AO2U7I2U3H5DEW62TAZQBZDFAFIOSNNMFSYAZOOTWMCK3CGBQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.965039048 (60h18m44.687602088s) [noise=true , address=45.91.116.5:4307 , piece_id=KJKQQT6QQHZOMWVFUHCAKIIY5WXPYQHL2XOZBJZKS5M3YLCXHSLQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.965338600 (60h18m44.68790163s) [noise=true , address=79.127.201.214:4289 , piece_id=Z7LQTNJAV77SGZVGCSNZ6VGVIM34EQ42FSFIWFS3RJHPEWEWXWIQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.965635338 (60h18m44.688198378s) [noise=true , address=79.127.201.214:4321 , piece_id=DF2ILKFA72AO4NMLNCWIXDDZF25YVCM6MIOUOZUXFKTRKOBGB2NQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.965931796 (60h18m44.688494836s) [noise=true , address=79.127.219.51:4210 , piece_id=AMXFGOUDEOGNMS7VJFIGU4A7U3QOMPYSC4LRTX2XC7CE6MJDNLLA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.966240756 (60h18m44.688803787s) [noise=true , address=79.127.205.235:4669 , piece_id=C5IA7TNMTBYFXWXSM3ZKSTFAE3I5D6JUMPYYI366P2RQLEOFCPOQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.966516544 (60h18m44.689079584s) [noise=true , address=79.127.201.215:4346 , piece_id=P3QVD6HME4T2KCAZSQFRXP6MZSOAQVHHJS3WGFWNTARENO7QNR4A, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.966778947 (60h18m44.689341987s) [noise=true , address=104.36.82.56:4237 , piece_id=2F3SYMVPIN2AZ6ZTIPQPQFBSAI37LFHPZ6HWVCASRC2RGJVR22WA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.967053752 (60h18m44.689616782s) [noise=true , address=79.127.205.245:4247 , piece_id=D7O3RT6CZL4Z5OLIMMH3ZPA7MXMWF6AIWBM36Q77UCC7U2TLOOJA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.967397630 (60h18m44.68996067s) [noise=true , address=79.127.220.101:4048 , piece_id=MPOSVPPT5VBGNRMYBFOIT3XVA6PR4ZBKQJCM4YUEDF3GJTGJ4MWA, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.967666955 (60h18m44.690229995s) [noise=true , address=104.36.82.52:4107 , piece_id=6QE7S6UMPSA7HOERNRUJQNUDNDVUNIG6HIC4XVIGNRBV2ENOU3IQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.967965427 (60h18m44.690528467s) [noise=true , address=104.36.82.47:4528 , piece_id=MDRZ2R7QLT7EJB3BPUHD4DZKR7ER4BQYZY3EOAHDGW7EKUIFUA4Q, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:34.968474551 (60h18m44.691037591s) [noise=true , address=104.36.82.13:4202 , piece_id=FS4UWHIQLX3WDTRZIWLVR36SLGR32DKUMFGXC4TZ2APVPXRBWDUQ, seg_pos={0 0}, upload=57853]: Uploading piece... Jul 26 23:54:35.883577688 (60h18m45.606140728s) [noise=true , address=79.127.220.101:4048 , piece_id=MPOSVPPT5VBGNRMYBFOIT3XVA6PR4ZBKQJCM4YUEDF3GJTGJ4MWA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:36.008798202 (60h18m45.731361242s) [noise=true , address=156.146.43.225:4449 , piece_id=3NNZDMPEHEG25BKQYLLFR66K57AZDBCVXFO3OXJGHX3UNFYVI6GA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:36.082760645 (60h18m45.805323685s) [upload=57853]: Next returned nil segment Jul 26 23:54:36.082779200 (60h18m45.805342241s) [upload=57853]: Waiting for error group managing segments... Jul 26 23:54:36.366675759 (60h18m46.089238789s) [noise=true , address=79.127.205.233:4647 , piece_id=OSFVPA6XPHZQHT74247OHBFSBSNEKMHESV4FOJC5ZXLJWBYYOA6Q, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:36.411084215 (60h18m46.133647255s) [noise=true , address=79.127.205.245:4247 , piece_id=D7O3RT6CZL4Z5OLIMMH3ZPA7MXMWF6AIWBM36Q77UCC7U2TLOOJA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:36.559004290 (60h18m46.28156732s) [noise=true , address=79.127.205.235:4669 , piece_id=C5IA7TNMTBYFXWXSM3ZKSTFAE3I5D6JUMPYYI366P2RQLEOFCPOQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:36.571488778 (60h18m46.294051818s) [noise=true , address=64.49.13.138:4482 , piece_id=666D7V6XKZIKTAXCEPV76IZ5UH6QIQM3QTQIIBXURD2ESFNWI6CA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:36.922725309 (60h18m46.645288349s) [noise=true , address=45.91.116.5:4307 , piece_id=KJKQQT6QQHZOMWVFUHCAKIIY5WXPYQHL2XOZBJZKS5M3YLCXHSLQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:37.082460740 (60h18m46.80502378s) [noise=true , address=79.127.201.214:4289 , piece_id=Z7LQTNJAV77SGZVGCSNZ6VGVIM34EQ42FSFIWFS3RJHPEWEWXWIQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:37.305708330 (60h18m47.02827136s) [noise=true , address=79.127.201.214:4321 , piece_id=DF2ILKFA72AO4NMLNCWIXDDZF25YVCM6MIOUOZUXFKTRKOBGB2NQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:39.612680522 (60h18m49.335243552s) [noise=true , address=156.146.43.225:4448 , piece_id=K77AO2U7I2U3H5DEW62TAZQBZDFAFIOSNNMFSYAZOOTWMCK3CGBQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:54:40.657784569 (60h18m50.380347649s) [noise=true , address=79.127.219.51:4210 , piece_id=AMXFGOUDEOGNMS7VJFIGU4A7U3QOMPYSC4LRTX2XC7CE6MJDNLLA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:55:12.822762147 (60h19m22.545325177s) [noise=true , address=79.127.201.215:4346 , piece_id=P3QVD6HME4T2KCAZSQFRXP6MZSOAQVHHJS3WGFWNTARENO7QNR4A, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:55:25.094326422 (60h19m34.816889513s) [noise=true , address=77.90.31.12:4596 , piece_id=D2ZHHBHBK6KEMKT5BTXOCPGWFXF3BEPKWRGNRO5ANLT6LKKBADHA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:55:33.583562151 (60h19m43.306125191s) [noise=true , address=77.90.31.10:4542 , piece_id=7HLQKYCD2KSJDGKQVRLEZYAJ4PVD4FLCUHJJ4MBL3N2S7TRAFPRQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: <nil> Jul 26 23:55:33.799838011 (60h19m43.522401041s) [noise=true , address=104.36.82.47:4528 , piece_id=MDRZ2R7QLT7EJB3BPUHD4DZKR7ER4BQYZY3EOAHDGW7EKUIFUA4Q, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 188.166.225.170:34570->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:33.799910660 (60h19m43.5224737s) [noise=true , address=104.36.82.56:4237 , piece_id=2F3SYMVPIN2AZ6ZTIPQPQFBSAI37LFHPZ6HWVCASRC2RGJVR22WA, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: rpc: dial tcp 104.36.82.56:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:33.799965375 (60h19m43.522528425s) [noise=true , address=104.36.82.52:4107 , piece_id=6QE7S6UMPSA7HOERNRUJQNUDNDVUNIG6HIC4XVIGNRBV2ENOU3IQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:33.799977918 (60h19m43.522540958s) [noise=true , address=104.36.82.13:4202 , piece_id=FS4UWHIQLX3WDTRZIWLVR36SLGR32DKUMFGXC4TZ2APVPXRBWDUQ, seg_pos={0 0}, upload=57853]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: write tcp 188.166.225.170:49138->104.36.82.13:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:33.799998879 (60h19m43.522561919s) [seg_pos={0 0}, upload=57853]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:55:34.033719075 (60h19m43.756282116s) [upload=57853]: Done uploading segments... Jul 26 23:55:34.033743592 (60h19m43.756306632s) [upload=57853]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.235685409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.239879251s FAIL (2025-07-26 23:55:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.23990576s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.813973766s FAIL (2025-07-26 23:55:26 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.814000828s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X.] duration: 7.420720276s FAIL (2025-07-26 23:55:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.420742149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 30.23395385s FAIL (2025-07-26 23:55:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 080301 (60h18m31.280643331s) [upload=57840]: Done uploading segments... Jul 26 23:54:21.558127612 (60h18m31.280690652s) [upload=57840]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:54:41.482044376 (60h18m51.204607406s) [upload=57856]: Starting upload Jul 26 23:54:41.482148315 (60h18m51.204711356s) [upload=57856]: Done starting upload Jul 26 23:54:41.482244520 (60h18m51.20480756s) [upload=57856]: Uploading segments... Jul 26 23:54:41.482380570 (60h18m51.20494361s) [upload=57856]: Got next segment. Inline: false Jul 26 23:54:41.482408975 (60h18m51.204972015s) [upload=57856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:41.482414044 (60h18m51.204977084s) [upload=57856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:41.717513470 (60h18m51.44007651s) [seg_pos={0 0}, upload=57856]: Begin upload segment... Jul 26 23:54:41.717575508 (60h18m51.440138548s) [seg_pos={0 0}, upload=57856]: Done begin upload segment. Jul 26 23:54:41.717624242 (60h18m51.440187312s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFNCOZGD5LJX4W6X3NZURM3DFE4WRQRTQHNVR7CXYZBVVQJDO4Q, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.717631896 (60h18m51.440194936s) [noise=true , address=104.36.82.15:4258 , piece_id=J3GOAH74ZLCTDFSVU3RAKTNRF3DSZLJAWXV25OGGLZS7K745Q2YQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.718078972 (60h18m51.440642012s) [noise=true , address=45.91.116.7:4011 , piece_id=R4Q6KJV5G6IX6YBVC7HKI7M4DPC2CDIZE7XAWGTHIL7BCFCZXSKA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.718225342 (60h18m51.440788382s) [noise=true , address=45.91.116.11:4119 , piece_id=O5XB3ITUI25C3BCW7ZVZIL3F34M4X6GVAGHQIMOI5ISHWJ4YGNNA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.718640025 (60h18m51.441203065s) [noise=true , address=45.91.116.6:4974 , piece_id=OMDPOU6W4W5INPQFQJYTFU7KQM4WFJAHKMRGQ4IGW7MKVOEEGXVQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.718986649 (60h18m51.441549689s) [noise=true , address=45.91.116.8:4048 , piece_id=RSHHZVS3A7E7JEXEG53Q4HGSJXPKYS6RVPBMUC5YDL6NSE6U6ZFQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.719387836 (60h18m51.441950876s) [noise=true , address=77.90.31.11:4553 , piece_id=4DVREWZR7N6CVFNSTR3MCSWKUOE5V7TF2QPPAGBAT57DIEM3BLOA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.719426330 (60h18m51.44198937s) [noise=true , address=45.91.116.13:4153 , piece_id=7WCTDE7HG7DUGHYVX2H47QOHNALVRFUARXDXJWQUB7QOLCBSIH2Q, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.719687289 (60h18m51.442250329s) [noise=true , address=104.36.82.16:4294 , piece_id=CU3RJEV6D4QZ4ZQASEPIB2FKV2C4GCMO7U27M34FLXTB5TLGYKHA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.719910255 (60h18m51.442473295s) [noise=true , address=77.90.31.17:4771 , piece_id=X4HIT5P5RB6IO62P6G5FU7RYP6I7YHZNCGVF6JCXADP4M2SYZBHA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.720212173 (60h18m51.442775213s) [noise=true , address=77.90.31.10:4520 , piece_id=FDMBEBZCJAGQW5SQMXJ6S2O7A5L4ACKPR77T23VMZ3CWEE57VFUQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.720458835 (60h18m51.443021875s) [noise=true , address=104.36.82.13:4186 , piece_id=A7K6VQHRHNHFX2J2HL6W3SCWFUF346RRRRYA4OXG4C7IFYH5CJLA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.720667124 (60h18m51.443230164s) [noise=true , address=104.36.82.11:4113 , piece_id=LRER4JCVDIEZO75CFIMGV7YYHFXXRFQDILUGZY3CODCZK4D5HGFQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.720967489 (60h18m51.443530529s) [noise=true , address=104.36.82.8:4971 , piece_id=AGTVABK7P7JPJ25LZORMHIIREZ6CWQW6S6DWIE2OYMSQHCFRHRNA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.721076137 (60h18m51.443639177s) [noise=true , address=45.91.116.10:4081 , piece_id=R34N5NN2DNYXXC2XKSSEF2YUVHCDQKP3WFLLV6S3QMXAI2CDAHNQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:41.721287580 (60h18m51.44385062s) [noise=true , address=104.36.82.16:4293 , piece_id=N5JI55LVYZUBVRSHUTUMO6TPRHFK5BY2Y7CE2DPBLN67EXP57PLA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:54:43.451589660 (60h18m53.1741527s) [upload=57856]: Next returned nil segment Jul 26 23:54:43.451612133 (60h18m53.174175173s) [upload=57856]: Waiting for error group managing segments... Jul 26 23:54:43.735990763 (60h18m53.458553793s) [noise=true , address=45.91.116.11:4119 , piece_id=O5XB3ITUI25C3BCW7ZVZIL3F34M4X6GVAGHQIMOI5ISHWJ4YGNNA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.747486970 (60h18m53.47005s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFNCOZGD5LJX4W6X3NZURM3DFE4WRQRTQHNVR7CXYZBVVQJDO4Q, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.825558764 (60h18m53.548121794s) [noise=true , address=45.91.116.6:4974 , piece_id=OMDPOU6W4W5INPQFQJYTFU7KQM4WFJAHKMRGQ4IGW7MKVOEEGXVQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.825869529 (60h18m53.548432569s) [noise=true , address=45.91.116.13:4153 , piece_id=7WCTDE7HG7DUGHYVX2H47QOHNALVRFUARXDXJWQUB7QOLCBSIH2Q, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.832026900 (60h18m53.55458994s) [noise=true , address=45.91.116.7:4011 , piece_id=R4Q6KJV5G6IX6YBVC7HKI7M4DPC2CDIZE7XAWGTHIL7BCFCZXSKA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.855503567 (60h18m53.578066597s) [noise=true , address=77.90.31.11:4553 , piece_id=4DVREWZR7N6CVFNSTR3MCSWKUOE5V7TF2QPPAGBAT57DIEM3BLOA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.883319027 (60h18m53.605882067s) [noise=true , address=45.91.116.8:4048 , piece_id=RSHHZVS3A7E7JEXEG53Q4HGSJXPKYS6RVPBMUC5YDL6NSE6U6ZFQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:54:43.894973466 (60h18m53.617536536s) [noise=true , address=45.91.116.10:4081 , piece_id=R34N5NN2DNYXXC2XKSSEF2YUVHCDQKP3WFLLV6S3QMXAI2CDAHNQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:55:07.620701292 (60h19m17.343264332s) [noise=true , address=77.90.31.10:4520 , piece_id=FDMBEBZCJAGQW5SQMXJ6S2O7A5L4ACKPR77T23VMZ3CWEE57VFUQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:55:10.839559745 (60h19m20.562122785s) [noise=true , address=104.36.82.13:4186 , piece_id=A7K6VQHRHNHFX2J2HL6W3SCWFUF346RRRRYA4OXG4C7IFYH5CJLA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839619049 (60h19m20.562182079s) [noise=true , address=104.36.82.8:4971 , piece_id=AGTVABK7P7JPJ25LZORMHIIREZ6CWQW6S6DWIE2OYMSQHCFRHRNA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:47284->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839653094 (60h19m20.562216134s) [noise=true , address=104.36.82.16:4293 , piece_id=N5JI55LVYZUBVRSHUTUMO6TPRHFK5BY2Y7CE2DPBLN67EXP57PLA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839658414 (60h19m20.562221454s) [noise=true , address=77.90.31.17:4771 , piece_id=X4HIT5P5RB6IO62P6G5FU7RYP6I7YHZNCGVF6JCXADP4M2SYZBHA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839711886 (60h19m20.562274926s) [noise=true , address=104.36.82.11:4113 , piece_id=LRER4JCVDIEZO75CFIMGV7YYHFXXRFQDILUGZY3CODCZK4D5HGFQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839745030 (60h19m20.56230807s) [noise=true , address=104.36.82.16:4294 , piece_id=CU3RJEV6D4QZ4ZQASEPIB2FKV2C4GCMO7U27M34FLXTB5TLGYKHA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839750110 (60h19m20.56231315s) [noise=true , address=104.36.82.15:4258 , piece_id=J3GOAH74ZLCTDFSVU3RAKTNRF3DSZLJAWXV25OGGLZS7K745Q2YQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:32866->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:10.839766440 (60h19m20.56232948s) [seg_pos={0 0}, upload=57856]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:55:11.071214981 (60h19m20.793778041s) [upload=57856]: Done uploading segments... Jul 26 23:55:11.071241472 (60h19m20.793804522s) [upload=57856]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.233973438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235035947s FAIL (2025-07-26 23:54:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23506392s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X] duration: 30.00051739s FAIL (2025-07-26 23:54:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000536696s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX..XXX............XXXX.....XXX..X.....X......X...XX..XX.X.] duration: 48.050990071s FAIL (2025-07-26 23:54:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.051000632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.797401688s FAIL (2025-07-26 23:54:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.797427938s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)