Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXX.XXX.X.X............................................] duration: 2.339740429s FAIL (2025-08-07 23:03:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.339763033s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X.........................................] duration: 6.664946477s FAIL (2025-08-07 23:03:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.664968538s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 642.265008ms target: https://us1.storj.io/login FAIL (2025-08-07 23:03:01 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 642.290866ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X....X.................X...................X.....] duration: 9.026707627s FAIL (2025-08-07 23:02:55 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 172565s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment... Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment. Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments... Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces. Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments... Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil> Duration:* 9.026736544s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...X...........................................] duration: 6.218216528s FAIL (2025-08-07 23:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.218273538s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [......XXXXXXXXX..X.X........................................] duration: 6.884801564s FAIL (2025-08-07 23:02:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.88482555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [X.XX..X.....................................................] duration: 1.930580457s FAIL (2025-08-07 23:02:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.930604222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 6.016521885s FAIL (2025-08-07 23:02:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.016548115s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX...X..........................................] duration: 6.390267053s FAIL (2025-08-07 23:02:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.390324493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......XXXXXXXXX..X.X.......................................] duration: 6.312533794s FAIL (2025-08-07 23:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.312554655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXX.X.................................................] duration: 1.897812466s FAIL (2025-08-07 23:02:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.897855117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 6.739656263s FAIL (2025-08-07 23:02:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.739685209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X..X...........................................] duration: 1.894792456s FAIL (2025-08-07 23:02:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.894812584s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 703.940153ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:02:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 703.968178ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 649.820974ms target: https://us1.storj.io/login FAIL (2025-08-07 23:02:05 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 649.84487ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXX.X........................................] duration: 5.613280308s FAIL (2025-08-07 23:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.613305646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXX.XXX.X.X...........................................] duration: 2.339276442s FAIL (2025-08-07 23:02:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.339298314s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX...X.........................................] duration: 5.948097558s FAIL (2025-08-07 23:01:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.948118528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [XX.XX..X....................................................] duration: 1.893898219s FAIL (2025-08-07 23:01:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.893928396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........X....X.................X...................X....] duration: 9.118309791s FAIL (2025-08-07 23:01:42 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 172565s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment... Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment. Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments... Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces. Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments... Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil> Duration:* 9.1183357s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 6.626561467s FAIL (2025-08-07 23:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.626585393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXX.X................................................] duration: 1.878096557s FAIL (2025-08-07 23:01:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.878116494s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 800.879609ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:01:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 800.903314ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X..X..........................................] duration: 1.964073276s FAIL (2025-08-07 23:01:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.964097723s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 646.613817ms target: https://us1.storj.io/login FAIL (2025-08-07 23:01:25 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 646.638304ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [XXX.XX..X...................................................] duration: 2.069712954s FAIL (2025-08-07 23:01:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.06981004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 5.951144548s FAIL (2025-08-07 23:01:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.951170367s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX...X........................................] duration: 6.231793454s FAIL (2025-08-07 23:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231829614s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXX.X.......................................] duration: 5.535970539s FAIL (2025-08-07 23:01:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.535991709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [........XXXXXXXXX..X.X......................................] duration: 7.725314003s FAIL (2025-08-07 23:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.72534502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X..X.........................................] duration: 1.969317727s FAIL (2025-08-07 23:01:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.969347104s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 763.600286ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:01:05 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 763.628691ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 5.82030759s FAIL (2025-08-07 23:00:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.820333119s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........X....X.................X...................X...] duration: 9.330933107s FAIL (2025-08-07 23:00:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 172565s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment... Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment. Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments... Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces. Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments... Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil> Duration:* 9.330956173s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXX.X.X..........................................] duration: 2.377637761s FAIL (2025-08-07 23:00:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.377686184s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXX..........................................] duration: 7.784319829s FAIL (2025-08-07 23:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.784347691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [XXXX.XX..X..................................................] duration: 1.907813373s FAIL (2025-08-07 23:00:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.907836387s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...X.......................................] duration: 6.020267959s FAIL (2025-08-07 23:00:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.020307315s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 5.776535382s FAIL (2025-08-07 23:00:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.776564509s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X...............................................] duration: 1.942585872s FAIL (2025-08-07 23:00:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.942614105s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 655.749022ms target: https://us1.storj.io/login FAIL (2025-08-07 23:00:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 655.770733ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X..X........................................] duration: 1.896902067s FAIL (2025-08-07 23:00:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.89693027s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........X....X.................X...................X..] duration: 7.781413366s FAIL (2025-08-07 23:00:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 172565s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment... Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment. Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece... Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments... Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil> Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces. Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments... Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil> Duration:* 7.781445217s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 719.312344ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:00:21 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 719.34038ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX...X......................................] duration: 6.168637969s FAIL (2025-08-07 23:00:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.168680991s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXX.X.X.........................................] duration: 2.395093137s FAIL (2025-08-07 23:00:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.395567827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X..............................................] duration: 1.915185106s FAIL (2025-08-07 23:00:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.915218852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXX.X......................................] duration: 5.674321612s FAIL (2025-08-07 23:00:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.674345007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXX.........................................] duration: 9.68206838s FAIL (2025-08-07 23:00:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.682108978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........XXXXXXXXX..X.X.....................................] duration: 8.174323387s FAIL (2025-08-07 22:59:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.174350227s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)