Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX.....X.XXX.............................] duration: 5.725481207s FAIL (2025-08-04 16:40:10 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.725514709s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................XX.X.XX.XX..X........................X.] duration: 5.717800087s FAIL (2025-08-04 16:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.717813472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.866658637s FAIL (2025-08-04 16:40:08 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.866682113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.70755208s FAIL (2025-08-04 16:40:06 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.707577489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.480568448s FAIL (2025-08-04 16:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.480591172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.031571554s FAIL (2025-08-04 16:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.031594409s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................XX.X.XX.XX..X........................X] duration: 30.000993232s FAIL (2025-08-04 16:40:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001050842s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.676270688s FAIL (2025-08-04 16:39:50 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.676295004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.317950538s FAIL (2025-08-04 16:39:30 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.317981137s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.429033469s FAIL (2025-08-04 16:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.42905493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX.....X.XXX............................] duration: 5.920245202s FAIL (2025-08-04 16:39:24 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.920274809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.65981731s FAIL (2025-08-04 16:39:10 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.659844123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.267989183s FAIL (2025-08-04 16:39:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.268012647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.969933778s FAIL (2025-08-04 16:39:07 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.969961983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.660175575s FAIL (2025-08-04 16:39:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.660201154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................XX.X.XX.XX..X........................] duration: 6.051111156s FAIL (2025-08-04 16:38:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.051136835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 8.466799501s FAIL (2025-08-04 16:38:54 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.466822996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX.....X.XXX...........................] duration: 6.013694789s FAIL (2025-08-04 16:38:38 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 6.013727202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.513626998s FAIL (2025-08-04 16:38:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.513653799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.531229179s FAIL (2025-08-04 16:38:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.531249889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.866819247s FAIL (2025-08-04 16:38:33 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.866842832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.875522092s FAIL (2025-08-04 16:38:32 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.875538655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.841724996s FAIL (2025-08-04 16:38:17 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.841756608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.700901927s FAIL (2025-08-04 16:38:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.700923248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.535609875s FAIL (2025-08-04 16:38:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.535627057s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................XX.X.XX.XX..X.......................] duration: 13.715198712s FAIL (2025-08-04 16:37:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.715219983s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.263312645s FAIL (2025-08-04 16:37:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.263331291s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX.....X.XXX..........................] duration: 7.809729906s FAIL (2025-08-04 16:37:38 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 7.809755404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.847381306s FAIL (2025-08-04 16:37:29 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.847407516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.993517692s FAIL (2025-08-04 16:37:27 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.993538703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.824360952s FAIL (2025-08-04 16:37:25 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.824388565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................XX.X.XX.XX..X......................] duration: 11.240682388s FAIL (2025-08-04 16:37:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.240707646s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.494147887s FAIL (2025-08-04 16:37:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.494168266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX.....X.XXX.........................] duration: 6.081025033s FAIL (2025-08-04 16:37:07 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 6.081045762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.791795555s FAIL (2025-08-04 16:37:04 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.791824159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.904529184s FAIL (2025-08-04 16:36:52 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.904553179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.86390207s FAIL (2025-08-04 16:36:47 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.863927907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.292669918s FAIL (2025-08-04 16:36:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.292693633s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................XX.X.XX.XX..X.....................] duration: 10.286217021s FAIL (2025-08-04 16:36:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.286238562s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.513906688s FAIL (2025-08-04 16:36:23 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.513937468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.901908254s FAIL (2025-08-04 16:36:18 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.90193725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.536849012s FAIL (2025-08-04 16:36:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.536936229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX.....X.XXX........................] duration: 6.148765421s FAIL (2025-08-04 16:35:57 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 6.1488007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.426799906s FAIL (2025-08-04 16:35:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.426821747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.548266862s FAIL (2025-08-04 16:35:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.548299424s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.823290534s FAIL (2025-08-04 16:35:41 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.823316373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................XX.X.XX.XX..X....................] duration: 6.395998033s FAIL (2025-08-04 16:35:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.396023423s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.839652755s FAIL (2025-08-04 16:35:31 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.839735562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX.....X.XXX.......................] duration: 5.793938769s FAIL (2025-08-04 16:35:24 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.793958708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.935663943s FAIL (2025-08-04 16:35:22 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.93568312s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)