Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.666802119s FAIL (2025-08-04 16:27:00 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.666831005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.372029321s FAIL (2025-08-04 16:26:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.372053367s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................XX.X.XX.XX..X...........] duration: 6.193462647s FAIL (2025-08-04 16:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.193482795s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.634910017s FAIL (2025-08-04 16:26:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.634945285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.845030903s FAIL (2025-08-04 16:26:46 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.845055179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XX.....X.XXX...........] duration: 5.817118825s FAIL (2025-08-04 16:26:39 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.817146317s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.318396742s FAIL (2025-08-04 16:26:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.318414847s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................................XX.X.XX.XX..X..........] duration: 6.271915665s FAIL (2025-08-04 16:26:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.271936896s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................................XX.X.XX.XX..X.........] duration: 6.26559456s FAIL (2025-08-04 16:26:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.265616189s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.487042901s FAIL (2025-08-04 16:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.487063008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.804275114s FAIL (2025-08-04 16:26:11 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.804307255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.935518859s FAIL (2025-08-04 16:26:09 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.935541805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.824107851s FAIL (2025-08-04 16:26:03 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.824135403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.303074015s FAIL (2025-08-04 16:26:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.303097992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.939426147s FAIL (2025-08-04 16:25:52 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.939446705s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................................XX.X.XX.XX..X........] duration: 6.178497636s FAIL (2025-08-04 16:25:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178518746s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.780015042s FAIL (2025-08-04 16:25:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.780272003s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XX.....X.XXX..........] duration: 5.921014367s FAIL (2025-08-04 16:25:35 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.921051147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.630995292s FAIL (2025-08-04 16:25:33 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.631018026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................XX.X.XX.XX..X.......] duration: 6.252369791s FAIL (2025-08-04 16:25:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.252403305s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.124342445s FAIL (2025-08-04 16:25:31 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.124370458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.644956682s FAIL (2025-08-04 16:25:12 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.644980489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.988987785s FAIL (2025-08-04 16:25:09 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.989031978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XX.....X.XXX.........] duration: 5.93086408s FAIL (2025-08-04 16:25:01 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.930888386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.671058988s FAIL (2025-08-04 16:25:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.671082622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.334712261s FAIL (2025-08-04 16:24:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.334735254s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.99730415s FAIL (2025-08-04 16:24:45 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.997324558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XX.....X.XXX........] duration: 5.970159223s FAIL (2025-08-04 16:24:44 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.970191383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.959306758s FAIL (2025-08-04 16:24:42 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.95933347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................XX.X.XX.XX..X......] duration: 5.979611821s FAIL (2025-08-04 16:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.979634335s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.455712244s FAIL (2025-08-04 16:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.455733716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.295214077s FAIL (2025-08-04 16:24:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.29523578s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.112658006s FAIL (2025-08-04 16:24:16 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.112683867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.859966566s FAIL (2025-08-04 16:24:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.859993167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XX.....X.XXX.......] duration: 5.992112498s FAIL (2025-08-04 16:23:40 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.992141634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.836621464s FAIL (2025-08-04 16:23:34 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.836648135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.415836036s FAIL (2025-08-04 16:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.415857348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.757229202s FAIL (2025-08-04 16:23:28 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.757255002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................XX.X.XX.XX..X.....] duration: 6.075345189s FAIL (2025-08-04 16:23:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.075359338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.868297842s FAIL (2025-08-04 16:23:14 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.868324663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.304943142s FAIL (2025-08-04 16:23:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.304966396s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.791764194s FAIL (2025-08-04 16:23:04 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.791792898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.771180661s FAIL (2025-08-04 16:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771219585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XX.....X.XXX......] duration: 5.888239624s FAIL (2025-08-04 16:23:00 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.88826377s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.389687896s FAIL (2025-08-04 16:22:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.389710628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.640663057s FAIL (2025-08-04 16:22:48 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.640709646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.903134572s FAIL (2025-08-04 16:22:45 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.903159641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XX.....X.XXX.....] duration: 5.833943791s FAIL (2025-08-04 16:22:26 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.833967424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.070473763s FAIL (2025-08-04 16:22:26 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.070496307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.644931224s FAIL (2025-08-04 16:22:24 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.645083615s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)