Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 6.440529791s FAIL (2025-07-26 21:11:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440737589s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.............X......X......X.....................X.....] duration: 6.635581456s FAIL (2025-07-26 21:11:14 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 6.635607055s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 6.299176055s FAIL (2025-07-26 21:11:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.299208088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 6.195088297s FAIL (2025-07-26 21:10:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.195118736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.............X......X......X.....................X....] duration: 7.530193341s FAIL (2025-07-26 21:10:06 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 7.530217117s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 6.161458915s FAIL (2025-07-26 21:10:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.161479866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.939216596s FAIL (2025-07-26 21:09:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.939242054s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 6.216239446s FAIL (2025-07-26 21:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21652431s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 5.991028643s FAIL (2025-07-26 21:09:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.991056798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.............X......X......X.....................X...] duration: 6.798746954s FAIL (2025-07-26 21:09:01 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 6.798773896s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 6.359131885s FAIL (2025-07-26 21:08:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.359150973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.901472511s FAIL (2025-07-26 21:08:51 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.901497668s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.............X......X......X.....................X..] duration: 6.191058228s FAIL (2025-07-26 21:08:20 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 6.191088586s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 6.243422647s FAIL (2025-07-26 21:08:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.243443889s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 5.750247299s FAIL (2025-07-26 21:08:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.750271174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.845035799s FAIL (2025-07-26 21:08:03 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.845063142s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 6.555510767s FAIL (2025-07-26 21:07:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.555550092s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.974348924s FAIL (2025-07-26 21:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.974372068s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.............X......X......X.....................X.] duration: 6.871202053s FAIL (2025-07-26 21:07:24 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 6.871232072s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.215852548s FAIL (2025-07-26 21:07:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.215899617s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.............X......X......X.....................X] duration: 6.016812309s FAIL (2025-07-26 21:06:46 UTC) orbiter warning: digest mismatch: after 56c2e09ba15d0e699d08c0d58f3749f0fbc9398fb5fe0914f19c8c549edc5ebd != before dc381fcf7a830470aaaf14a12393caa67b6283bf63bfbc61c099113c1510e5f0 Duration: 6.016861835s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.845008697s FAIL (2025-07-26 21:06:45 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.845031991s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 6.37678524s FAIL (2025-07-26 21:06:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.37680589s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 6.45968378s FAIL (2025-07-26 21:06:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.459714829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.892777686s FAIL (2025-07-26 21:06:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.892855824s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 6.279506895s FAIL (2025-07-26 21:05:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.279524738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.............X......X......X.....................] duration: 7.065529912s FAIL (2025-07-26 21:05:47 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 7.065545682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.866470591s FAIL (2025-07-26 21:05:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.866501861s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 6.811710188s FAIL (2025-07-26 21:05:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.811737421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.88715222s FAIL (2025-07-26 21:05:24 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.887182818s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 5.981750683s FAIL (2025-07-26 21:05:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.981771532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.011515058s FAIL (2025-07-26 21:04:54 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.011537161s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 6.221141251s FAIL (2025-07-26 21:04:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.221170206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.............X......X......X....................] duration: 6.670222415s FAIL (2025-07-26 21:04:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.67025118s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.979248095s FAIL (2025-07-26 21:04:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.979269125s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.............X......X......X...................] duration: 6.024840418s FAIL (2025-07-26 21:04:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.024868691s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 6.930329824s FAIL (2025-07-26 21:04:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.930352319s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 6.657085693s FAIL (2025-07-26 21:04:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.657107233s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.925352392s FAIL (2025-07-26 21:04:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.925381057s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.910840769s FAIL (2025-07-26 21:03:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.910871306s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 7.143149554s FAIL (2025-07-26 21:03:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.143175714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............X......X......X..................] duration: 6.852210504s FAIL (2025-07-26 21:03:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.852236995s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 6.21339486s FAIL (2025-07-26 21:03:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.213438303s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.980767151s FAIL (2025-07-26 21:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980787911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.858685135s FAIL (2025-07-26 21:03:02 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.858714001s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.............X.............X......X......X.................] duration: 6.297450006s FAIL (2025-07-26 21:02:54 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.297490885s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 6.541937048s FAIL (2025-07-26 21:02:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.541967716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.............X.............X......X......X................] duration: 9.02730947s FAIL (2025-07-26 21:02:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 9.027332593s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 6.32455394s FAIL (2025-07-26 21:02:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== iece_id=JIGNC43VKD3YVARU6BHSPMLAMT3FRSKLSPNAURBYNTFANBMCJ7RA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:38056->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255340599 (56h52m56.977903639s) [noise=true , address=45.91.116.14:4189 , piece_id=H6XUSMWDVUJXANZVSQETB7Y44KEBB574DCOHEXWBIBOJXXH5Y7OA, seg_pos={0 0}, upload=54593]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: rpc: dial tcp 45.91.116.14:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:47.255384323 (56h52m56.977947363s) [seg_pos={0 0}, upload=54593]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 20:28:47.486510951 (56h52m57.209073991s) [upload=54593]: Done uploading segments... Jul 26 20:28:47.486551077 (56h52m57.209114107s) [upload=54593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 20:28:50.713181364 (56h53m0.435744404s) [upload=54599]: Starting upload Jul 26 20:28:50.713248432 (56h53m0.435811472s) [upload=54599]: Done starting upload Jul 26 20:28:50.713312314 (56h53m0.435875354s) [upload=54599]: Uploading segments... Jul 26 20:28:50.713452573 (56h53m0.436015603s) [upload=54599]: Got next segment. Inline: false Jul 26 20:28:50.713476478 (56h53m0.436039519s) [upload=54599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:50.713480777 (56h53m0.436043817s) [upload=54599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:50.949390366 (56h53m0.671953466s) [seg_pos={0 0}, upload=54599]: Begin upload segment... Jul 26 20:28:50.949444028 (56h53m0.672007078s) [seg_pos={0 0}, upload=54599]: Done begin upload segment. Jul 26 20:28:50.949500426 (56h53m0.672063466s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949629433 (56h53m0.672192463s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.949977509 (56h53m0.672540539s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950235172 (56h53m0.672798212s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950257304 (56h53m0.672820344s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950548682 (56h53m0.673111722s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950751770 (56h53m0.6733148s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950764715 (56h53m0.673327755s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.950977603 (56h53m0.673540643s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951011087 (56h53m0.673574127s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951350225 (56h53m0.673913265s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951743598 (56h53m0.674306628s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.951870891 (56h53m0.674433931s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.952532227 (56h53m0.675095257s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.953348769 (56h53m0.675911809s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:50.956236154 (56h53m0.678799184s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Uploading piece... Jul 26 20:28:54.584558914 (56h53m4.307121954s) [upload=54599]: Next returned nil segment Jul 26 20:28:54.584581508 (56h53m4.307144548s) [upload=54599]: Waiting for error group managing segments... Jul 26 20:28:55.375834954 (56h53m5.098398034s) [noise=true , address=77.90.31.12:4590 , piece_id=ARHJG7RVPKZVH6CUXFOPEDKB2AJIPMFAJVG25Q2TK6LHD4IBWWWA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:55.754417801 (56h53m5.476980841s) [noise=true , address=77.90.31.11:4554 , piece_id=BUYLPY7NLDS5FSYPDDXYKUZ33PNEHXN4BHO7XY27JJTP7YGHAZVQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.120567801 (56h53m5.843130841s) [noise=true , address=77.90.31.17:4772 , piece_id=S43R2DCSBAEYSAL3BNCQ7ALNLHPSZCA7VZ5YQHOSMIJQAYM2DECQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.147135805 (56h53m5.869698845s) [noise=true , address=77.90.31.18:4805 , piece_id=P7KQ5SZ2ISZOCS3MCVWX2YPOXNG24K5ZTAARIDVQ7PAZXVGBMNFQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:56.217572301 (56h53m5.940135341s) [noise=true , address=77.90.31.13:4628 , piece_id=A6XWIROEGOZXJC25NNJAJGC6NNFZV6K4GDLWECXWF6P5H4GIMHAQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.106221031 (56h53m7.828784101s) [noise=true , address=77.90.31.12:4589 , piece_id=GO5HZYFCBDNKSE3DMK6BYNRQK6QSHTFN25NPHGOVRX3UCPLDGYNQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:28:58.696825733 (56h53m8.419388764s) [noise=true , address=77.90.31.17:4771 , piece_id=IL4AGLMJUO4FOI2DFWTVESJVBXQESOPBPCQPY72IATQXJ4KZ6BCA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.838197092 (56h53m18.560760122s) [noise=true , address=45.91.116.14:4191 , piece_id=OYBQLD4KTXNQ4YAPBXJPRREOZNM3IH6GDUAJWMIQUNGMPHEKDNSA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.840921915 (56h53m18.563484955s) [noise=true , address=45.91.116.4:4939 , piece_id=4I4GPIZZLQ3BSRZSMJ2NL6QXXYCZ7NJZY3EA6GH6K2OLA7PET55Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.944779827 (56h53m18.667342867s) [noise=true , address=45.91.116.14:4190 , piece_id=OWBIICTMFAUPD7JZHZSQ32PL723LKT6JIXQTUAJEYIQO625JBX7Q, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:08.968949992 (56h53m18.691513032s) [noise=true , address=45.91.116.8:4047 , piece_id=3F4C2BQNUZVNEN2AISGVHD3KML5VNDIAIXDWOQMPST5TNBDMTSMA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:09.074863428 (56h53m18.797426468s) [noise=true , address=45.91.116.7:4011 , piece_id=KHVS7FKAGCRNTYVWEYGMXRSICPH3KOC4JXQJERWRC6X7PBRWJ24A, seg_pos={0 0}, upload=54599]: Done uploading piece. err: <nil> Jul 26 20:29:20.062079499 (56h53m29.784642579s) [noise=true , address=104.36.82.7:4910 , piece_id=SEHRW22C4QUPD2VO2CFPXITZPKWPHF45656ZH6L4O57ZQCP7JYZA, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062272147 (56h53m29.784835218s) [noise=true , address=104.36.82.13:4187 , piece_id=JVAKXUM3WAGXXIM4EC23DE43HXDAGPRLZQH2FKQI44EQ76633DTQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062359615 (56h53m29.784922655s) [noise=true , address=104.36.82.9:4005 , piece_id=OJUNB6IYWDBPXHKJYNFYMSEE3A6Y6JNYX44FBRJKTJPHJPCCU6KQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062433816 (56h53m29.784996856s) [noise=true , address=104.36.82.10:4078 , piece_id=G366YAGZHU2V64PJP7KGFDSAVE5GH3IOXZPXFVNO5QO6UDBDMNQQ, seg_pos={0 0}, upload=54599]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: context deadline exceeded; context deadline exceeded Jul 26 20:29:20.062447683 (56h53m29.785010723s) [seg_pos={0 0}, upload=54599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 20:29:20.289763289 (56h53m30.012326339s) [upload=54599]: Done uploading segments... Jul 26 20:29:20.289790922 (56h53m30.012353962s) [upload=54599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.324574941s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 6.357888734s FAIL (2025-07-26 21:02:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.357909934s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)