Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXX..XX..X..............................XXX...........] duration: 5.819662484s FAIL (2025-08-07 22:37:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.819688474s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X............................XXXXXXXXXXXXXXXXX.X...........] duration: 5.536685682s FAIL (2025-08-07 22:37:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.536720518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXX.X..X.............] duration: 1.898604952s FAIL (2025-08-07 22:37:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.89862546s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 230.161541ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:37:23 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 230.185728ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X................................XXXXXXXXX..X.X.........] duration: 6.235557938s FAIL (2025-08-07 22:37:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.235584188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXX.XX..X............] duration: 1.878659493s FAIL (2025-08-07 22:37:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.878684019s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..X....................] duration: 759.604949ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:37:06 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 759.633604ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXX............] duration: 9.380923673s FAIL (2025-08-07 22:37:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.381025888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXX.X............] duration: 1.881991488s FAIL (2025-08-07 22:37:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.882011596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXX..XX..X..............................XXX..........] duration: 5.608850746s FAIL (2025-08-07 22:37:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.608880803s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.018653643s FAIL (2025-08-07 22:36:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.018680975s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXX.XXX.X.X.........] duration: 2.341127061s FAIL (2025-08-07 22:36:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.341149382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....................X..............X.........X....X.......] duration: 7.147067s FAIL (2025-08-07 22:36:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 7.147084273s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXX.XX..X...........] duration: 1.840126278s FAIL (2025-08-07 22:36:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.840148089s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXX...X.........] duration: 5.870961895s FAIL (2025-08-07 22:36:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.870982684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.152184821s FAIL (2025-08-07 22:36:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.152208777s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXX.X...........] duration: 1.952460895s FAIL (2025-08-07 22:36:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.952483949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXX.XX..X..........] duration: 2.083288572s FAIL (2025-08-07 22:36:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.08331395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..X............................XXXXXXXXXXXXXXXXX.X..........] duration: 5.613690532s FAIL (2025-08-07 22:36:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.613712946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXX.X..X............] duration: 1.907256415s FAIL (2025-08-07 22:36:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.907281453s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 242.854727ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:36:20 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 242.870617ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXX.XXX.X.X........] duration: 2.331341035s FAIL (2025-08-07 22:36:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.33135931s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X................................XXXXXXXXX..X.X........] duration: 6.359511261s FAIL (2025-08-07 22:36:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.359537211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.136439186s FAIL (2025-08-07 22:36:06 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.136455626s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 268.100869ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:36:04 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 268.125206ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X..X............................XXXXXXXXXXXXXXXXX.X.........] duration: 5.611884139s FAIL (2025-08-07 22:36:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.611906691s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration: 714.083495ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:36:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 714.110125ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXX...X........] duration: 6.154370817s FAIL (2025-08-07 22:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.154397777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXX...........] duration: 8.68083907s FAIL (2025-08-07 22:35:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.680878265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXX..XX..X..............................XXX.........] duration: 5.743241074s FAIL (2025-08-07 22:35:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.743288645s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 18 out of 60 is failed (threshold=3) [X...............................XXXXXXXXXXXXXXXX.X..........] duration: 1.9717245s FAIL (2025-08-07 22:35:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.97175631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....................X..............X.........X....X......] duration: 9.129331401s FAIL (2025-08-07 22:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 9.129359153s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXX.XXX.X.X.......] duration: 2.717178144s FAIL (2025-08-07 22:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717191709s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.X..X............................XXXXXXXXXXXXXXXXX.X........] duration: 5.649530726s FAIL (2025-08-07 22:35:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.649553097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXX.X..X...........] duration: 1.869724989s FAIL (2025-08-07 22:35:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.869752182s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X................................XXXXXXXXXXXXXXXXX..........] duration: 6.53336126s FAIL (2025-08-07 22:35:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.533395124s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXX.XXX.X.X......] duration: 2.217311225s FAIL (2025-08-07 22:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.217365659s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXX.XX..X.........] duration: 1.869992955s FAIL (2025-08-07 22:35:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.870017832s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..X..................] duration: 789.030591ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:35:12 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 789.063353ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 18 out of 60 is failed (threshold=3) [.X...............................XXXXXXXXXXXXXXXX.X.........] duration: 1.973957283s FAIL (2025-08-07 22:35:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.97400737s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 286.971995ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:35:09 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 286.997604ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXX...X.......] duration: 6.301188435s FAIL (2025-08-07 22:35:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.301210298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.....X................................XXXXXXXXX..X.X.......] duration: 7.380147297s FAIL (2025-08-07 22:35:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.380168388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....................X..............X.........X....X.....] duration: 7.39819584s FAIL (2025-08-07 22:34:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 7.398212582s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.327269115s FAIL (2025-08-07 22:34:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.32728743s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..XX..X..............................XXX........] duration: 5.933867873s FAIL (2025-08-07 22:34:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.933892009s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXX.XXX.X.X.....] duration: 2.25667043s FAIL (2025-08-07 22:34:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.256718513s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X................................XXXXXXXXXXXXXXXXX.........] duration: 6.651274074s FAIL (2025-08-07 22:34:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.651291377s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXX...X......] duration: 6.059829663s FAIL (2025-08-07 22:34:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.059850311s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..X..X............................XXXXXXXXXXXXXXXXX.X.......] duration: 5.624603206s FAIL (2025-08-07 22:34:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.624629126s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)