Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.................X.....X.........X.X.....X.......X......X] duration: 30.001848603s trace: https://jaeger.dev.storj.io/trace/259d01d688000700 FAIL (2025-11-22 14:45:30 UTC) orbiter warning: context deadline exceeded Duration: 30.001873963s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..................X........X.......XX..X..XX..X.X.........XX] duration: 5.002070651s trace: https://jaeger.dev.storj.io/trace/54280af9336f517e FAIL (2025-11-22 14:45:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002096028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX.X....] duration: 14.392856472s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:45:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 14.392890979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XX.X...] duration: 24.680374373s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:44:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 24.680403369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.................X.....X.........X.X.....X.......X......] duration: 20.371630901s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:44:48 UTC) orbiter warning: context deadline exceeded Duration: 20.371658695s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.................X.....X.........X.X.....X.......X.....] duration: 23.431726688s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:44:14 UTC) orbiter warning: context deadline exceeded Duration: 23.43175404s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...................X........X.......XX..X..XX..X.X.........X] duration: 5.001986181s trace: https://jaeger.dev.storj.io/trace/7e230ba78fd5efe3 FAIL (2025-11-22 14:44:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002013793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XX.X..] duration: 15.061860701s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:44:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 15.061888803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.................X.....X.........X.X.....X.......X....] duration: 17.280546132s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:43:42 UTC) orbiter warning: context deadline exceeded Duration: 17.28057592s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX.X.] duration: 15.544959534s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:43:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 15.544995764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 30.547794374s trace: https://jaeger.dev.storj.io/trace/2098099cdbf22f5b FAIL (2025-11-22 14:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ress=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533160124 (318h38m49.084630575s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.533955108 (318h38m49.085425569s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534408512 (318h38m49.085879003s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.534918874 (318h38m49.086389325s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535369011 (318h38m49.086839472s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.535799019 (318h38m49.08726947s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.536169635 (318h38m49.087640096s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:50.537636228 (318h38m49.089106689s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Uploading piece... Nov 22 14:42:52.464943215 (318h38m51.016413686s) [upload=280627]: Next returned nil segment Nov 22 14:42:52.464961469 (318h38m51.01643194s) [upload=280627]: Waiting for error group managing segments... Nov 22 14:42:52.897385742 (318h38m51.448856203s) [noise=true , address=45.91.116.14:4190 , piece_id=BA6BACRBCPZBOJKTDKZSFMN35UV6W4FV33JU2PR7XMSRUA5Z52QA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.046847795 (318h38m51.598318246s) [noise=true , address=77.90.31.13:4627 , piece_id=GTHKTBIV73LNWLPDTL2PKBNIC7VZYOYK7G7ERSZXXSLD5ESEM24Q, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:42:53.186925158 (318h38m51.738395619s) [noise=true , address=45.91.116.4:4938 , piece_id=ADBIO2OZ6VVGITYEUULMYZDVXPO2LHG5W2FDSB3Q5KVVYYTL7ATQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:05.929989912 (318h39m4.481460373s) [noise=true , address=145.79.227.20:4438 , piece_id=IIQHQ3GKVL6YWXRYKXAXMPGZQU7LSQQW5BYJKPETAVPXMSTDM7NQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:14.692577362 (318h39m13.244047833s) [noise=true , address=145.79.227.23:4545 , piece_id=CC7YROPFVVKV6HA4VNV6IWVH3PIMJ37YGTLGIPX65OX6UYYTV7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: <nil> Nov 22 14:43:18.282495199 (318h39m16.83396568s) [noise=true , address=145.79.227.13:4804 , piece_id=ERBDFEJD7IJMGEVZFCO7N26UQLUUU7MEPLI5P7SV32FG2P7L4ZOQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 14:43:18.282705419 (318h39m16.83417588s) [noise=true , address=145.79.227.9:4367 , piece_id=M7ZCUIJ6EISGCTND462MAVP27I7AEO323BOFO7V6ZA4VC252L6IA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35686->145.79.227.9:4367: use of closed network connection Nov 22 14:43:18.282734765 (318h39m16.834205216s) [noise=true , address=145.79.227.18:4007 , piece_id=ONFLDVV3DIZIRV7ZVT5TOH4S5NDHBVWZ6YOE2RJ2QWWMZIHVN6AA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49488->145.79.227.18:4007: use of closed network connection Nov 22 14:43:18.282822593 (318h39m16.834293054s) [noise=true , address=145.79.227.22:4509 , piece_id=NDKKBLT6TCAFSK4C2SJQXMW5RATRE3M34VKKMICI6SUGNFFGT5SA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41416->145.79.227.22:4509: use of closed network connection Nov 22 14:43:18.282853250 (318h39m16.834323711s) [noise=true , address=145.79.227.22:4511 , piece_id=TJORE67AD3XQEMVPWSTAZ4VIGEY4N2QL6UTHX2QL66H7MCNAQ5CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47720->145.79.227.22:4511: use of closed network connection Nov 22 14:43:18.282918334 (318h39m16.834388785s) [noise=true , address=145.79.227.14:4840 , piece_id=3DQXISRXP6PQPZIPYCFA5U3AH3I7XSUQQ7OYA2IUH2TGLYBSF4QQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58068->145.79.227.14:4840: use of closed network connection Nov 22 14:43:18.282946428 (318h39m16.834416879s) [noise=true , address=145.79.227.21:4473 , piece_id=QKI2QDUHLIUDLEYR3Z2T3IMWZXEZARLNCEIM3H25AGW4W4BSP2CQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44386->145.79.227.21:4473: use of closed network connection Nov 22 14:43:18.283049174 (318h39m16.834519635s) [noise=true , address=145.79.227.16:4909 , piece_id=KAWAKHTGYORPJAERAHQE2P66IWO262Y35NIABWNKQ2WUTT4KH2MQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.16:4909: use of closed network connection Nov 22 14:43:18.283143564 (318h39m16.834614045s) [noise=true , address=145.79.227.15:4873 , piece_id=GFCBZPEBLKOZDBNFS5Q5LRU2T4ETW6HDT77K3C4WZOL4M5MFJ7OA, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59742->145.79.227.15:4873: use of closed network connection Nov 22 14:43:18.283420130 (318h39m16.834890601s) [noise=true , address=145.79.227.30:4761 , piece_id=IEZBETWEMNU5RJQKSTUWTIWYP2LNUHI54DB63G4HVQQWAEBPDNHQ, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->145.79.227.30:4761: use of closed network connection Nov 22 14:43:18.283443234 (318h39m16.834913725s) [noise=true , address=145.79.227.11:4730 , piece_id=4LOAOMJ5T6RPGQLU47MY3QSDJJARGFRIHD2YCKFSPTDH3X6FNJ6A, seg_pos={0 0}, upload=280627]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:38836->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:43:18.283513477 (318h39m16.834983938s) [seg_pos={0 0}, upload=280627]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 22 14:43:18.827845814 (318h39m17.379316275s) [upload=280627]: Done uploading segments... Nov 22 14:43:18.827896642 (318h39m17.379367103s) [upload=280627]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.547963727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....................X........X.......XX..X..XX..X.X.........] duration: 3.570700344s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:42:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570805405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.................X.....X.........X.X.....X.......X...] duration: 25.591704813s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:42:39 UTC) orbiter warning: context deadline exceeded Duration: 25.591737394s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....................X........X.......XX..X..XX..X.X........] duration: 3.617129923s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:42:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.617142938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......................X........X.......XX..X..XX..X.X.......] duration: 4.490230034s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:42:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.490251074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 15.587203905s trace: https://jaeger.dev.storj.io/trace/5427605889d9cdd4 FAIL (2025-11-22 14:42:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== v 22 14:41:03.116119543 (318h37m1.667590014s) [upload=280597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:41:03.718921296 (318h37m2.270391757s) [seg_pos={0 0}, upload=280597]: Begin upload segment... Nov 22 14:41:03.719037687 (318h37m2.270508148s) [seg_pos={0 0}, upload=280597]: Done begin upload segment. Nov 22 14:41:03.719096008 (318h37m2.27056647s) [noise=true , address=145.79.227.17:4970 , piece_id=E4T2B4RHS2OD2JWJ2CEMNHRAEQSIIJ6EATVNCBVGHIGK6BYFOOQA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719117419 (318h37m2.27058789s) [noise=true , address=77.90.31.11:4554 , piece_id=FPIPE7HZHEFXMCUJ24U7XS2TRMVMI33F6WEQDK7TOE7OT4DPGHVA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719185649 (318h37m2.27065611s) [noise=true , address=145.79.227.8:4329 , piece_id=Q2AHDEUAIDDGXYESHFKPKGPLSQ7XME6MLWVDZLJFK7CZ6F7KBTMA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719855404 (318h37m2.271325865s) [noise=true , address=145.79.227.25:4617 , piece_id=GCRTLTMOTSBYXZGX7KTTXM2XHPHM74Q6YBNKZZ2RRTWFTD4TY54Q, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719862017 (318h37m2.271332478s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720242502 (318h37m2.271712963s) [noise=true , address=145.79.227.22:4511 , piece_id=WISK5BFIX6RHJEKVIPIFZVM43EKAZQEA7H5XYQ7RKV342XIJRA3A, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720300892 (318h37m2.271771353s) [noise=true , address=45.91.116.6:4975 , piece_id=5R65F5RFFM5ZCMCMLKKEVHVGWXUWKESITMTZ6S6COEJ5O4IKH2GQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720581677 (318h37m2.272052138s) [noise=true , address=45.91.116.13:4153 , piece_id=SSCLEM63ZEAR4XIBEVTYDZ6NK5GWAU36J2IUOTBG2Q6RH2T6RPCA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720817496 (318h37m2.272287957s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720896867 (318h37m2.272367328s) [noise=true , address=145.79.227.28:4725 , piece_id=7YFM4V3RQOWP3F4SFP3NREHQ7DPYNM33LSWOS2TPLQQZUKDGC4TQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721201167 (318h37m2.272671628s) [noise=true , address=77.90.31.12:4589 , piece_id=N3G5YEJYTSEC2F4L3V5GVQQXWDAVVRPOTDKCNLJUXHSMYOBLZSUA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721318230 (318h37m2.272788691s) [noise=true , address=145.79.227.20:4438 , piece_id=CWPIOKOGCNIWRQ7EDPK3XDGTHRAZPCCCJZF5TMNX5NPPDYM3S4HQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721900278 (318h37m2.273370739s) [noise=true , address=145.79.227.20:4439 , piece_id=54BXGQTLRWJLXT6U54R6SAZRVVDYR2PAN3UE46VZVVBLG3SH5WYQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.722542692 (318h37m2.274013153s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.722846319 (318h37m2.27431678s) [noise=true , address=77.90.31.19:4843 , piece_id=IBA3HV5RU7QMY76JW2S2CQ62QNYHOV55FLP6LTBAHSDZMKOGJIIA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.723323267 (318h37m2.274793739s) [noise=true , address=145.79.227.18:4007 , piece_id=7CPQEGWRYS3XBG6TD43TJIN42EKALZUOEXNVPTOH777VHJLUSFPQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:05.652494846 (318h37m4.203965307s) [upload=280597]: Next returned nil segment Nov 22 14:41:05.652512681 (318h37m4.203983142s) [upload=280597]: Waiting for error group managing segments... Nov 22 14:41:06.133997525 (318h37m4.685467986s) [noise=true , address=77.90.31.19:4843 , piece_id=IBA3HV5RU7QMY76JW2S2CQ62QNYHOV55FLP6LTBAHSDZMKOGJIIA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.222588888 (318h37m4.774059349s) [noise=true , address=45.91.116.6:4975 , piece_id=5R65F5RFFM5ZCMCMLKKEVHVGWXUWKESITMTZ6S6COEJ5O4IKH2GQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.338902049 (318h37m4.89037251s) [noise=true , address=45.91.116.13:4153 , piece_id=SSCLEM63ZEAR4XIBEVTYDZ6NK5GWAU36J2IUOTBG2Q6RH2T6RPCA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.632671559 (318h37m5.18414202s) [noise=true , address=77.90.31.12:4589 , piece_id=N3G5YEJYTSEC2F4L3V5GVQQXWDAVVRPOTDKCNLJUXHSMYOBLZSUA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.638138479 (318h37m5.18960893s) [noise=true , address=77.90.31.11:4554 , piece_id=FPIPE7HZHEFXMCUJ24U7XS2TRMVMI33F6WEQDK7TOE7OT4DPGHVA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:15.337545723 (318h37m13.889016184s) [noise=true , address=145.79.227.8:4329 , piece_id=Q2AHDEUAIDDGXYESHFKPKGPLSQ7XME6MLWVDZLJFK7CZ6F7KBTMA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:15.870446421 (318h37m14.421916882s) [noise=true , address=145.79.227.20:4438 , piece_id=CWPIOKOGCNIWRQ7EDPK3XDGTHRAZPCCCJZF5TMNX5NPPDYM3S4HQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:16.845485971 (318h37m15.396956432s) [noise=true , address=145.79.227.17:4970 , piece_id=E4T2B4RHS2OD2JWJ2CEMNHRAEQSIIJ6EATVNCBVGHIGK6BYFOOQA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:17.453786435 (318h37m16.005256896s) [noise=true , address=145.79.227.20:4439 , piece_id=54BXGQTLRWJLXT6U54R6SAZRVVDYR2PAN3UE46VZVVBLG3SH5WYQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:18.118436158 (318h37m16.669906619s) [noise=true , address=145.79.227.22:4511 , piece_id=WISK5BFIX6RHJEKVIPIFZVM43EKAZQEA7H5XYQ7RKV342XIJRA3A, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:21.814692914 (318h37m20.366163375s) [noise=true , address=145.79.227.25:4617 , piece_id=GCRTLTMOTSBYXZGX7KTTXM2XHPHM74Q6YBNKZZ2RRTWFTD4TY54Q, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:22.340905357 (318h37m20.892375838s) [noise=true , address=145.79.227.18:4007 , piece_id=7CPQEGWRYS3XBG6TD43TJIN42EKALZUOEXNVPTOH777VHJLUSFPQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:24.109981405 (318h37m22.661451876s) [noise=true , address=145.79.227.28:4725 , piece_id=7YFM4V3RQOWP3F4SFP3NREHQ7DPYNM33LSWOS2TPLQQZUKDGC4TQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:24.110005982 (318h37m22.661476443s) [seg_pos={0 0}, upload=280597]: Segment reached optimal threshold of 13 pieces. Nov 22 14:41:24.110229998 (318h37m22.661700459s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection Nov 22 14:41:24.110238765 (318h37m22.661709226s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110384603 (318h37m22.661855064s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection Nov 22 14:41:24.110400753 (318h37m22.661871215s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110499311 (318h37m22.661969772s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection Nov 22 14:41:24.110509300 (318h37m22.661979761s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110531152 (318h37m22.662001613s) [seg_pos={0 0}, upload=280597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 14:41:24.110694664 (318h37m22.662165125s) [upload=280597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 14:41:24.110702158 (318h37m22.662172619s) [upload=280597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 14:41:24.748210347 (318h37m23.299680828s) [upload=280597]: Done uploading segments... Nov 22 14:41:24.748235335 (318h37m23.299705796s) [upload=280597]: Upload finished. err: <nil> Duration:* 15.58723258s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.................X.....X.........X.X.....X.......X..] duration: 6.577992178s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:42:03 UTC) orbiter warning: context deadline exceeded Duration: 6.578017638s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.................X.....X.........X.X.....X.......X.] duration: 7.52347464s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:41:32 UTC) orbiter warning: context deadline exceeded Duration: 7.52349611s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.002472067s trace: https://jaeger.dev.storj.io/trace/5427605889d9cdd4 FAIL (2025-11-22 14:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== v 22 14:41:03.116119543 (318h37m1.667590014s) [upload=280597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:41:03.718921296 (318h37m2.270391757s) [seg_pos={0 0}, upload=280597]: Begin upload segment... Nov 22 14:41:03.719037687 (318h37m2.270508148s) [seg_pos={0 0}, upload=280597]: Done begin upload segment. Nov 22 14:41:03.719096008 (318h37m2.27056647s) [noise=true , address=145.79.227.17:4970 , piece_id=E4T2B4RHS2OD2JWJ2CEMNHRAEQSIIJ6EATVNCBVGHIGK6BYFOOQA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719117419 (318h37m2.27058789s) [noise=true , address=77.90.31.11:4554 , piece_id=FPIPE7HZHEFXMCUJ24U7XS2TRMVMI33F6WEQDK7TOE7OT4DPGHVA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719185649 (318h37m2.27065611s) [noise=true , address=145.79.227.8:4329 , piece_id=Q2AHDEUAIDDGXYESHFKPKGPLSQ7XME6MLWVDZLJFK7CZ6F7KBTMA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719855404 (318h37m2.271325865s) [noise=true , address=145.79.227.25:4617 , piece_id=GCRTLTMOTSBYXZGX7KTTXM2XHPHM74Q6YBNKZZ2RRTWFTD4TY54Q, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.719862017 (318h37m2.271332478s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720242502 (318h37m2.271712963s) [noise=true , address=145.79.227.22:4511 , piece_id=WISK5BFIX6RHJEKVIPIFZVM43EKAZQEA7H5XYQ7RKV342XIJRA3A, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720300892 (318h37m2.271771353s) [noise=true , address=45.91.116.6:4975 , piece_id=5R65F5RFFM5ZCMCMLKKEVHVGWXUWKESITMTZ6S6COEJ5O4IKH2GQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720581677 (318h37m2.272052138s) [noise=true , address=45.91.116.13:4153 , piece_id=SSCLEM63ZEAR4XIBEVTYDZ6NK5GWAU36J2IUOTBG2Q6RH2T6RPCA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720817496 (318h37m2.272287957s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.720896867 (318h37m2.272367328s) [noise=true , address=145.79.227.28:4725 , piece_id=7YFM4V3RQOWP3F4SFP3NREHQ7DPYNM33LSWOS2TPLQQZUKDGC4TQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721201167 (318h37m2.272671628s) [noise=true , address=77.90.31.12:4589 , piece_id=N3G5YEJYTSEC2F4L3V5GVQQXWDAVVRPOTDKCNLJUXHSMYOBLZSUA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721318230 (318h37m2.272788691s) [noise=true , address=145.79.227.20:4438 , piece_id=CWPIOKOGCNIWRQ7EDPK3XDGTHRAZPCCCJZF5TMNX5NPPDYM3S4HQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.721900278 (318h37m2.273370739s) [noise=true , address=145.79.227.20:4439 , piece_id=54BXGQTLRWJLXT6U54R6SAZRVVDYR2PAN3UE46VZVVBLG3SH5WYQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.722542692 (318h37m2.274013153s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.722846319 (318h37m2.27431678s) [noise=true , address=77.90.31.19:4843 , piece_id=IBA3HV5RU7QMY76JW2S2CQ62QNYHOV55FLP6LTBAHSDZMKOGJIIA, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:03.723323267 (318h37m2.274793739s) [noise=true , address=145.79.227.18:4007 , piece_id=7CPQEGWRYS3XBG6TD43TJIN42EKALZUOEXNVPTOH777VHJLUSFPQ, seg_pos={0 0}, upload=280597]: Uploading piece... Nov 22 14:41:05.652494846 (318h37m4.203965307s) [upload=280597]: Next returned nil segment Nov 22 14:41:05.652512681 (318h37m4.203983142s) [upload=280597]: Waiting for error group managing segments... Nov 22 14:41:06.133997525 (318h37m4.685467986s) [noise=true , address=77.90.31.19:4843 , piece_id=IBA3HV5RU7QMY76JW2S2CQ62QNYHOV55FLP6LTBAHSDZMKOGJIIA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.222588888 (318h37m4.774059349s) [noise=true , address=45.91.116.6:4975 , piece_id=5R65F5RFFM5ZCMCMLKKEVHVGWXUWKESITMTZ6S6COEJ5O4IKH2GQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.338902049 (318h37m4.89037251s) [noise=true , address=45.91.116.13:4153 , piece_id=SSCLEM63ZEAR4XIBEVTYDZ6NK5GWAU36J2IUOTBG2Q6RH2T6RPCA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.632671559 (318h37m5.18414202s) [noise=true , address=77.90.31.12:4589 , piece_id=N3G5YEJYTSEC2F4L3V5GVQQXWDAVVRPOTDKCNLJUXHSMYOBLZSUA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:06.638138479 (318h37m5.18960893s) [noise=true , address=77.90.31.11:4554 , piece_id=FPIPE7HZHEFXMCUJ24U7XS2TRMVMI33F6WEQDK7TOE7OT4DPGHVA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:15.337545723 (318h37m13.889016184s) [noise=true , address=145.79.227.8:4329 , piece_id=Q2AHDEUAIDDGXYESHFKPKGPLSQ7XME6MLWVDZLJFK7CZ6F7KBTMA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:15.870446421 (318h37m14.421916882s) [noise=true , address=145.79.227.20:4438 , piece_id=CWPIOKOGCNIWRQ7EDPK3XDGTHRAZPCCCJZF5TMNX5NPPDYM3S4HQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:16.845485971 (318h37m15.396956432s) [noise=true , address=145.79.227.17:4970 , piece_id=E4T2B4RHS2OD2JWJ2CEMNHRAEQSIIJ6EATVNCBVGHIGK6BYFOOQA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:17.453786435 (318h37m16.005256896s) [noise=true , address=145.79.227.20:4439 , piece_id=54BXGQTLRWJLXT6U54R6SAZRVVDYR2PAN3UE46VZVVBLG3SH5WYQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:18.118436158 (318h37m16.669906619s) [noise=true , address=145.79.227.22:4511 , piece_id=WISK5BFIX6RHJEKVIPIFZVM43EKAZQEA7H5XYQ7RKV342XIJRA3A, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:21.814692914 (318h37m20.366163375s) [noise=true , address=145.79.227.25:4617 , piece_id=GCRTLTMOTSBYXZGX7KTTXM2XHPHM74Q6YBNKZZ2RRTWFTD4TY54Q, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:22.340905357 (318h37m20.892375838s) [noise=true , address=145.79.227.18:4007 , piece_id=7CPQEGWRYS3XBG6TD43TJIN42EKALZUOEXNVPTOH777VHJLUSFPQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:24.109981405 (318h37m22.661451876s) [noise=true , address=145.79.227.28:4725 , piece_id=7YFM4V3RQOWP3F4SFP3NREHQ7DPYNM33LSWOS2TPLQQZUKDGC4TQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: <nil> Nov 22 14:41:24.110005982 (318h37m22.661476443s) [seg_pos={0 0}, upload=280597]: Segment reached optimal threshold of 13 pieces. Nov 22 14:41:24.110229998 (318h37m22.661700459s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36196->145.79.227.30:4762: use of closed network connection Nov 22 14:41:24.110238765 (318h37m22.661709226s) [noise=true , address=145.79.227.30:4762 , piece_id=22BIVNZLGHNOXOVVRI74PJUXXL4TOHDW3ILBSLWOMRFQG5GJSJIQ, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110384603 (318h37m22.661855064s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53492->145.79.227.14:4838: use of closed network connection Nov 22 14:41:24.110400753 (318h37m22.661871215s) [noise=true , address=145.79.227.14:4838 , piece_id=Q43ICGM3CFV3X75PW4OHPZJRK5MUFUL26S643RJ27TOSG6YPKRHA, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110499311 (318h37m22.661969772s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->145.79.227.23:4545: use of closed network connection Nov 22 14:41:24.110509300 (318h37m22.661979761s) [noise=true , address=145.79.227.23:4545 , piece_id=AN4TCDPTTR4A733GNZXS6AUFSESF2H7CIMBP3HLBKD5EEU3IF7OQ, seg_pos={0 0}, upload=280597]: Long tail cancellation due to optimal threshold Nov 22 14:41:24.110531152 (318h37m22.662001613s) [seg_pos={0 0}, upload=280597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 14:41:24.110694664 (318h37m22.662165125s) [upload=280597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 14:41:24.110702158 (318h37m22.662172619s) [upload=280597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 14:41:24.748210347 (318h37m23.299680828s) [upload=280597]: Done uploading segments... Nov 22 14:41:24.748235335 (318h37m23.299705796s) [upload=280597]: Upload finished. err: <nil> Duration:* 30.002494128s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.................X.....X.........X.X.....X.......X] duration: 30.001474663s trace: https://jaeger.dev.storj.io/trace/14c7f4b5b4c2308e FAIL (2025-11-22 14:41:25 UTC) orbiter warning: context deadline exceeded Duration: 30.001501877s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......................X........X.......XX..X..XX..X.X......] duration: 5.001046359s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:41:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001075656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.60924615s trace: https://jaeger.dev.storj.io/trace/6e53e8619bb1683f FAIL (2025-11-22 14:40:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4SDV6RQCHTYMBKVUKP4EK76X4KWOPUW5GZU66ALOYSNTEX6Q, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.527694872 (318h36m25.079165333s) [noise=true , address=145.79.227.30:4763 , piece_id=QH66YNGQBAP4LIGFLSOGUQCZUWC3BRHHHZ7O5WIX5CN3FEBUEGBA, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.527780826 (318h36m25.079251277s) [noise=true , address=145.79.227.16:4909 , piece_id=GJM5ADX322MGF6AKLUMDAQ32GZQ7C6OA7BBY7HYGHF4IKCWCZ5IQ, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.528116114 (318h36m25.079586585s) [noise=true , address=77.90.31.19:4842 , piece_id=Z62FCOUVZ2PBVT4YEZBF272MAW6UK6RC7SF6M7C6WZW7NJNW4N7A, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.528238307 (318h36m25.079708808s) [noise=true , address=145.79.227.17:4969 , piece_id=VIGBR3HQEMU2LL3ZOJNFJNMTKAX4DB3CIKF6CVV37GOHKB2EGGQQ, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.528469968 (318h36m25.079940429s) [noise=true , address=145.79.227.20:4437 , piece_id=5PWWNLXCWZLU3ZLP4MKNNZ4QGXJDCRDPBFZVNCHH773UTKZ4DDPA, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.528718992 (318h36m25.080189443s) [noise=true , address=145.79.227.23:4546 , piece_id=7QLSZ3ZWH6XLS4YDKWX7I72H42APEXEGISW22IWRE7BFJC5JITUQ, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.528760862 (318h36m25.080231323s) [noise=true , address=145.79.227.15:4875 , piece_id=FV6ALZ4DK7XOSUMDG344EAP7XUCQK7FFH5LAVYHA2MEQP5FUZU3Q, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.529050654 (318h36m25.080521115s) [noise=true , address=145.79.227.27:4690 , piece_id=Z6J6FR5MNOM64JUNGOAYKY6KQHXCXZBIQ5KWNQM2RWEQI3HB4OEA, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:26.529162367 (318h36m25.080632818s) [noise=true , address=77.90.31.11:4554 , piece_id=5UX6C2ATQPEU4DABJERYY2H3T34BYI6C7AHUNWZ3LBI7CUNULW4A, seg_pos={0 0}, upload=280592]: Uploading piece... Nov 22 14:40:28.540770941 (318h36m27.092241392s) [upload=280592]: Next returned nil segment Nov 22 14:40:28.540812389 (318h36m27.09228285s) [upload=280592]: Waiting for error group managing segments... Nov 22 14:40:28.968970940 (318h36m27.520441402s) [noise=true , address=77.90.31.19:4842 , piece_id=Z62FCOUVZ2PBVT4YEZBF272MAW6UK6RC7SF6M7C6WZW7NJNW4N7A, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:29.026274717 (318h36m27.577745288s) [noise=true , address=77.90.31.15:4699 , piece_id=RIFD67RCI54F5HWFFDAHZGSIYUPL2PQRE5KLCHBRC2PUQZX44ECQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:29.138453335 (318h36m27.689923796s) [noise=true , address=45.91.116.9:4262 , piece_id=CYIA4SDV6RQCHTYMBKVUKP4EK76X4KWOPUW5GZU66ALOYSNTEX6Q, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:29.165259309 (318h36m27.71672977s) [noise=true , address=77.90.31.11:4554 , piece_id=5UX6C2ATQPEU4DABJERYY2H3T34BYI6C7AHUNWZ3LBI7CUNULW4A, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:30.148702772 (318h36m28.700173223s) [noise=true , address=45.91.116.7:4011 , piece_id=N6Q36NSLU2PSIBSPFLKNF5I5MDYM6LZQOO5PRS564SU6ZWXUVJOQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:38.839929086 (318h36m37.391399547s) [noise=true , address=145.79.227.27:4690 , piece_id=Z6J6FR5MNOM64JUNGOAYKY6KQHXCXZBIQ5KWNQM2RWEQI3HB4OEA, seg_pos={0 0}, upload=280592]: Done uploading piece. err: <nil> Nov 22 14:40:54.129834562 (318h36m52.681305063s) [noise=true , address=145.79.227.20:4437 , piece_id=5PWWNLXCWZLU3ZLP4MKNNZ4QGXJDCRDPBFZVNCHH773UTKZ4DDPA, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:36542->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:36542->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36542->145.79.227.20:4437: use of closed network connection Nov 22 14:40:54.130006409 (318h36m52.68147687s) [noise=true , address=145.79.227.14:4839 , piece_id=VN6SLWMM7Z62RTDKQFEWMMEYQSOXGBAONHLCJ6MGCQKLOKL52HTQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:55556->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:55556->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55556->145.79.227.14:4839: use of closed network connection Nov 22 14:40:54.130177063 (318h36m52.681647524s) [noise=true , address=145.79.227.17:4969 , piece_id=VIGBR3HQEMU2LL3ZOJNFJNMTKAX4DB3CIKF6CVV37GOHKB2EGGQQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: write tcp 188.166.225.170:60566->145.79.227.17:4969: use of closed network connection; write tcp 188.166.225.170:60566->145.79.227.17:4969: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60566->145.79.227.17:4969: use of closed network connection Nov 22 14:40:54.130290610 (318h36m52.681761071s) [noise=true , address=145.79.227.13:4804 , piece_id=SQSL3KQ43DK6EZ2V6L6BRU4G7P6XY4B66U637WTJ6YBDWPALTVSA, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52806->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52806->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52806->145.79.227.13:4804: use of closed network connection Nov 22 14:40:54.130381923 (318h36m52.681852384s) [noise=true , address=145.79.227.30:4763 , piece_id=QH66YNGQBAP4LIGFLSOGUQCZUWC3BRHHHZ7O5WIX5CN3FEBUEGBA, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:35052->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:35052->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35052->145.79.227.30:4763: use of closed network connection Nov 22 14:40:54.130470933 (318h36m52.681941394s) [noise=true , address=145.79.227.15:4875 , piece_id=FV6ALZ4DK7XOSUMDG344EAP7XUCQK7FFH5LAVYHA2MEQP5FUZU3Q, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:52654->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:52654->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52654->145.79.227.15:4875: use of closed network connection Nov 22 14:40:54.130561837 (318h36m52.682032288s) [noise=true , address=145.79.227.23:4546 , piece_id=7QLSZ3ZWH6XLS4YDKWX7I72H42APEXEGISW22IWRE7BFJC5JITUQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:45828->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:45828->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45828->145.79.227.23:4546: use of closed network connection Nov 22 14:40:54.130649353 (318h36m52.682119814s) [noise=true , address=145.79.227.10:4401 , piece_id=AD6FTWHFLD24UHLYONY6CLJGDY4DP5CDPLBSTLIT4EC4LMKIBK2Q, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:35542->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:35542->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35542->145.79.227.10:4401: use of closed network connection Nov 22 14:40:54.130755615 (318h36m52.682226076s) [noise=true , address=145.79.227.16:4909 , piece_id=GJM5ADX322MGF6AKLUMDAQ32GZQ7C6OA7BBY7HYGHF4IKCWCZ5IQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:55528->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:55528->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55528->145.79.227.16:4909: use of closed network connection Nov 22 14:40:54.131155718 (318h36m52.682626179s) [noise=true , address=145.79.227.22:4510 , piece_id=L6UB4EGLZCYNQJEIB47NJ64AEPXVDBQGOVI6PERAZ4TZQRLD62KQ, seg_pos={0 0}, upload=280592]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:58178->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:40:54.131191015 (318h36m52.682661476s) [seg_pos={0 0}, upload=280592]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 22 14:40:54.737922290 (318h36m53.289392752s) [upload=280592]: Done uploading segments... Nov 22 14:40:54.737969500 (318h36m53.289439961s) [upload=280592]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.609272501s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........................X........X.......XX..X..XX..X.X.....] duration: 4.590356834s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:40:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.590399645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.................X.....X.........X.X.....X.......] duration: 22.640650054s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:40:15 UTC) orbiter warning: context deadline exceeded Duration: 22.640681545s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........................X........X.......XX..X..XX..X.X....] duration: 3.692176117s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:39:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.692207485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..........................X........X.......XX..X..XX..X.X...] duration: 3.570658205s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:39:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570700085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.................X.....X.........X.X.....X......] duration: 18.51852251s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:39:03 UTC) orbiter warning: context deadline exceeded Duration: 18.518566734s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...........................X........X.......XX..X..XX..X.X..] duration: 3.52550838s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:38:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525568857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.................X.....X.........X.X.....X.....] duration: 6.530095956s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:38:04 UTC) orbiter warning: context deadline exceeded Duration: 6.530157804s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [............................X........X.......XX..X..XX..X.X.] duration: 3.446117516s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:37:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446151662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.................X.....X.........X.X.....X....] duration: 6.513417427s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:37:04 UTC) orbiter warning: context deadline exceeded Duration: 6.513449218s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.............................X........X.......XX..X..XX..X.X] duration: 5.001607218s trace: https://jaeger.dev.storj.io/trace/17b97dc602055889 FAIL (2025-11-22 14:36:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001643166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.................X.....X.........X.X.....X...] duration: 21.119241028s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:36:08 UTC) orbiter warning: context deadline exceeded Duration: 21.11925801s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.................X.....X.........X.X.....X..] duration: 25.569045165s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:35:46 UTC) orbiter warning: context deadline exceeded Duration: 25.569091673s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............................X........X.......XX..X..XX..X.] duration: 4.573720325s trace: https://jaeger.dev.storj.io/trace/608efc8bc98fb8c0 FAIL (2025-11-22 14:35:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.573779768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X.................X.....X.........X.X.....X.] duration: 29.445672198s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:35:09 UTC) orbiter warning: context deadline exceeded Duration: 29.445687546s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............................X........X.......XX..X..XX..X] duration: 5.000533614s trace: https://jaeger.dev.storj.io/trace/608efc8bc98fb8c0 FAIL (2025-11-22 14:34:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000557269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X.................X.....X.........X.X.....X] duration: 30.001705959s trace: https://jaeger.dev.storj.io/trace/62bc846f5c6ed8d8 FAIL (2025-11-22 14:34:40 UTC) orbiter warning: context deadline exceeded Duration: 30.001729153s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................................X........X.......XX..X..XX..] duration: 3.571933434s trace: https://jaeger.dev.storj.io/trace/650c15d118c53d41 FAIL (2025-11-22 14:34:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571949734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................................X........X.......XX..X..XX.] duration: 3.563493291s trace: https://jaeger.dev.storj.io/trace/650c15d118c53d41 FAIL (2025-11-22 14:34:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563511567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.................X.....X.........X.X.....] duration: 6.2878903s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:33:23 UTC) orbiter warning: context deadline exceeded Duration: 6.28803689s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..................................X........X.......XX..X..XX] duration: 5.002573653s trace: https://jaeger.dev.storj.io/trace/650c15d118c53d41 FAIL (2025-11-22 14:33:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002587501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.................X.....X.........X.X....] duration: 20.556434829s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:32:49 UTC) orbiter warning: context deadline exceeded Duration: 20.556462963s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................................X........X.......XX..X..X] duration: 5.001723905s trace: https://jaeger.dev.storj.io/trace/b91750e957967b4 FAIL (2025-11-22 14:32:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001755555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.................X.....X.........X.X...] duration: 8.706830274s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:31:22 UTC) orbiter warning: context deadline exceeded Duration: 8.706844311s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.................X.....X.........X.X..] duration: 22.361037027s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:31:13 UTC) orbiter warning: context deadline exceeded Duration: 22.361064398s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................................X........X.......XX..X..] duration: 5.001363249s trace: https://jaeger.dev.storj.io/trace/2693a67e1a232dfb FAIL (2025-11-22 14:30:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001407483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X.................X.....X.........X.X.] duration: 20.418986714s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:30:19 UTC) orbiter warning: context deadline exceeded Duration: 20.419016279s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................................X........X.......XX..X.] duration: 3.541075576s trace: https://jaeger.dev.storj.io/trace/2693a67e1a232dfb FAIL (2025-11-22 14:30:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541103248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.................X.....X.........X.X] duration: 30.001771709s trace: https://jaeger.dev.storj.io/trace/1b8c5948762083be FAIL (2025-11-22 14:29:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001795704s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)