Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.023242653s FAIL (2025-08-07 23:52:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.0232716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X...X..X.X...........................................] duration: 1.858155864s FAIL (2025-08-07 23:52:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.858177025s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..X.XXXX.X............................................] duration: 6.398336234s FAIL (2025-08-07 23:52:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.398361162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....................X..............X........X.....] duration: 7.273133027s FAIL (2025-08-07 23:52:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.273159198s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........XXXX.X..XX.......................................] duration: 6.587065373s FAIL (2025-08-07 23:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.58708973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration: 1.996672029s FAIL (2025-08-07 23:51:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.996695696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........XXXXXX.X.X.........................................] duration: 5.749378993s FAIL (2025-08-07 23:51:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.749400324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X......X................................X.............] duration: 1.84659296s FAIL (2025-08-07 23:51:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.846633258s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 650.594027ms target: https://us1.storj.io/login FAIL (2025-08-07 23:51:34 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 650.620155ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 1.993333161s FAIL (2025-08-07 23:51:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.99335856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........XXXXXX.X.X........................................] duration: 5.639198403s FAIL (2025-08-07 23:51:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.639209545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.643419873s FAIL (2025-08-07 23:51:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.643448116s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XX.X.........................................] duration: 6.521292033s FAIL (2025-08-07 23:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.521317752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..X.XXXX.X...........................................] duration: 6.749652324s FAIL (2025-08-07 23:51:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.749675689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........X...X..X.X..........................................] duration: 1.874450087s FAIL (2025-08-07 23:51:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.874475374s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration: 2.042987552s FAIL (2025-08-07 23:50:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.043014995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....................X..............X........X....] duration: 9.22716254s FAIL (2025-08-07 23:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 9.227191857s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............XXXX.X..XX......................................] duration: 6.286562602s FAIL (2025-08-07 23:50:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.28660335s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 652.822308ms target: https://us1.storj.io/login FAIL (2025-08-07 23:50:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 652.838028ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.81492621s FAIL (2025-08-07 23:50:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.814954966s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X.X...................................] duration: 2.222100575s FAIL (2025-08-07 23:50:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.222118157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..X.XXXX.X..........................................] duration: 6.797077079s FAIL (2025-08-07 23:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.797102197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 1.842711415s FAIL (2025-08-07 23:50:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.842774285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X......X................................X............] duration: 1.849250617s FAIL (2025-08-07 23:50:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.849276906s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....................X..............X........X...] duration: 6.722316442s FAIL (2025-08-07 23:50:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 6.722341038s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX.X........................................] duration: 7.954353517s FAIL (2025-08-07 23:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.954377843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............XXXX.X..XX.....................................] duration: 6.384501623s FAIL (2025-08-07 23:50:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.384560817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration: 1.916560937s FAIL (2025-08-07 23:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.916578952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........X...X..X.X.........................................] duration: 1.895777853s FAIL (2025-08-07 23:50:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.895803783s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........XXXXXX.X.X.......................................] duration: 6.532970033s FAIL (2025-08-07 23:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.532992026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....................X..............X........X..] duration: 7.659471751s FAIL (2025-08-07 23:50:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.659500014s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..X.XXXX.X.........................................] duration: 6.621187544s FAIL (2025-08-07 23:49:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.621215409s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............XXXX.X..XX....................................] duration: 6.242559612s FAIL (2025-08-07 23:49:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.242580571s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X.X..................................] duration: 2.400681766s FAIL (2025-08-07 23:49:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.400704389s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............XXXXXX.X.X......................................] duration: 5.589324013s FAIL (2025-08-07 23:49:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.589341777s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 651.051078ms target: https://us1.storj.io/login FAIL (2025-08-07 23:49:44 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.078242ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X......X................................X...........] duration: 2.072141391s FAIL (2025-08-07 23:49:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 2.072168773s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..X.XXXX.X........................................] duration: 6.483542418s FAIL (2025-08-07 23:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.483567126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.703774055s FAIL (2025-08-07 23:49:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.703800336s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............XXXX...........................................] duration: 1.886757435s FAIL (2025-08-07 23:49:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.886783985s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X.X.................................] duration: 2.383799653s FAIL (2025-08-07 23:49:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.383826054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........X...X..X.X........................................] duration: 1.904056311s FAIL (2025-08-07 23:49:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.904085267s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........X......X................................X..........] duration: 1.847334137s FAIL (2025-08-07 23:49:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.847374735s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X..X.XXXX.X.......................................] duration: 6.265823082s FAIL (2025-08-07 23:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.265854752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............XXXX..........................................] duration: 2.105584217s FAIL (2025-08-07 23:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.105603072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XX.X.......................................] duration: 10.134422574s FAIL (2025-08-07 23:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 10.134457241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration: 1.859516506s FAIL (2025-08-07 23:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.859545503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............XXXXXX.X.X.....................................] duration: 5.594260971s FAIL (2025-08-07 23:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.594283745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.....................X..............X........X.] duration: 9.141084515s FAIL (2025-08-07 23:48:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 9.141106727s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............XXXX.X..XX...................................] duration: 6.307391293s FAIL (2025-08-07 23:48:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.30740476s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)