Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.............X.............X......X......X...] duration: 5.716646286s FAIL (2025-07-26 20:53:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.716667828s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 6.433071823s FAIL (2025-07-26 20:53:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.433111689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.828796181s FAIL (2025-07-26 20:53:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.828822179s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 6.264547543s FAIL (2025-07-26 20:53:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.26457211s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.862095322s FAIL (2025-07-26 20:52:45 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.862125158s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..............X.............X.............X......X......X..] duration: 6.780982173s FAIL (2025-07-26 20:52:27 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.781018463s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 9.65062121s FAIL (2025-07-26 20:52:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.650661536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 6.922339173s FAIL (2025-07-26 20:52:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.922374472s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 6.31316322s FAIL (2025-07-26 20:51:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.313184961s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.883325062s FAIL (2025-07-26 20:51:48 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.883354678s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..............X.............X.............X......X......X.] duration: 5.982240008s FAIL (2025-07-26 20:51:22 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.982264515s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.427716126s FAIL (2025-07-26 20:50:57 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 3.427737566s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 6.270669964s FAIL (2025-07-26 20:50:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.270698009s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 6.193185584s FAIL (2025-07-26 20:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.193204292s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..............X.............X.............X......X......X] duration: 4.252731827s FAIL (2025-07-26 20:50:41 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 4.252754431s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 6.562861781s FAIL (2025-07-26 20:50:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.562887101s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 6.464361889s FAIL (2025-07-26 20:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.464384232s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.079463075s FAIL (2025-07-26 20:50:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.079493322s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..............X.............X.............X......X......] duration: 6.943361743s FAIL (2025-07-26 20:49:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.94341199s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 5.976229319s FAIL (2025-07-26 20:49:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.97628806s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 6.469766598s FAIL (2025-07-26 20:48:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.469828298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.891020966s FAIL (2025-07-26 20:48:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.891045724s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..............X.............X.............X......X.....] duration: 6.756758253s FAIL (2025-07-26 20:48:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.756789082s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 7.034816025s FAIL (2025-07-26 20:48:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.034835791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.879122167s FAIL (2025-07-26 20:48:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.879152155s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.845089584s FAIL (2025-07-26 20:48:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.845115234s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 6.079576117s FAIL (2025-07-26 20:47:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.079593971s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 6.636264095s FAIL (2025-07-26 20:47:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.636311085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.966467413s FAIL (2025-07-26 20:47:43 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.966493514s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..............X.............X.............X......X....] duration: 6.136061872s FAIL (2025-07-26 20:47:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.136085616s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 6.152059667s FAIL (2025-07-26 20:47:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.152085166s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 6.464375001s FAIL (2025-07-26 20:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.464398876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.858233329s FAIL (2025-07-26 20:46:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.858251064s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 6.028061014s FAIL (2025-07-26 20:46:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.028076373s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..............X.............X.............X......X...] duration: 5.94158647s FAIL (2025-07-26 20:46:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.941613212s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.858777682s FAIL (2025-07-26 20:46:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.858805204s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 5.820702875s FAIL (2025-07-26 20:46:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.820730658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..............X.............X.............X......X..] duration: 9.16531112s FAIL (2025-07-26 20:45:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.165337139s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.042681204s FAIL (2025-07-26 20:45:40 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.042707885s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 7.689120863s FAIL (2025-07-26 20:45:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.689140251s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 5.877871406s FAIL (2025-07-26 20:45:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.877895023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 6.037937679s FAIL (2025-07-26 20:45:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.03796939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..............X.............X.............X......X.] duration: 6.037566668s FAIL (2025-07-26 20:45:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.037597156s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.874957949s FAIL (2025-07-26 20:44:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.874978047s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 6.149800427s FAIL (2025-07-26 20:44:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.149814384s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.911032541s FAIL (2025-07-26 20:44:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.911055014s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 7.747983016s FAIL (2025-07-26 20:44:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.748012571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..............X.............X.............X......X] duration: 30.242257565s FAIL (2025-07-26 20:44:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24227627s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.841107161s FAIL (2025-07-26 20:43:37 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.841135716s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 6.167784363s FAIL (2025-07-26 20:43:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167807138s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)