Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 862139 reports (Page 11738 of 17243)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.108954667s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:53:22 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1.108981308s core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X...X.......XX.................] duration: 3.559801536s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:53:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.55982537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............XXX..........XX......X.....................XX..] duration: 5.898933011s trace: https://jaeger.dev.storj.io/trace/5e0d0482a82d8f09 FAIL (2025-12-08 07:53:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.898962038s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 695.636036ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:53:08 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 695.663247ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................X...X.......XX................] duration: 3.398455229s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398484034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...........X.] duration: 5.949965887s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 07:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.949982739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...........X] duration: 40.54963237s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 07:52:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 40.549670112s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................X...X.......XX...............] duration: 3.462700783s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:52:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462815912s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 474.298145ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:52:19 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.33275ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.............XXX..........XX......X.....................XX.] duration: 5.919878054s trace: https://jaeger.dev.storj.io/trace/5e0d0482a82d8f09 FAIL (2025-12-08 07:52:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.919894034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X.............XXX..........XX......X.....................XX] duration: 1m0.001722301s trace: https://jaeger.dev.storj.io/trace/5e0d0482a82d8f09 FAIL (2025-12-08 07:52:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001753379s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 859.371992ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:51:54 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 859.399274ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 469.727441ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:51:22 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.762479ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................X...X.......XX..............] duration: 3.524545297s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524575515s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 747.925857ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:51:13 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 747.961335ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.125043452s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:51:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.125070974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X.............XXX..........XX......X.....................X] duration: 1m0.002231063s trace: https://jaeger.dev.storj.io/trace/13e0cc6e47f2a87c FAIL (2025-12-08 07:51:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00237641s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 467.795001ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:50:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.820461ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.680446287s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.680473979s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................X...X.......XX.............] duration: 3.554907535s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:50:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554936208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.264796144s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.264827936s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 703.354291ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:50:08 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 703.380423ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 693.458139ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:49:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 693.48507ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X.............XXX..........XX......X.....................] duration: 5.867033483s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:49:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.867061896s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 470.641092ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:49:40 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.663505ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................X...X.......XX............] duration: 3.396900516s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:49:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.396926255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.812938726s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.812971026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X.............XXX..........XX......X....................] duration: 5.743620126s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.743645745s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 693.177367ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:48:38 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 693.18934ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m3.497959423s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 07:48:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1m3.497986805s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................X...X.......XX...........] duration: 3.402378838s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402402905s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 472.407704ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:48:31 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.424996ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X.............XXX..........XX......X...................] duration: 5.561048822s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:48:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.56106848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.921897718s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:48:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.921914509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X.............XXX..........XX......X..................] duration: 5.619638416s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:47:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.619658164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.147028727s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:47:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.147060326s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................X...X.......XX..........] duration: 3.422872722s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:47:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422899833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.798011249s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:47:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.79804335s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......X.............XXX..........XX......X.................] duration: 5.93148749s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:47:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.931520132s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 468.167361ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:46:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.196966ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.694111086s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.694136232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.899911233s trace: https://jaeger.dev.storj.io/trace/10deaed4f0583195 FAIL (2025-12-08 07:46:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.899943143s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................................X...X.......XX.........] duration: 3.423464227s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:46:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423488444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X.............XXX..........XX......X................] duration: 5.703282438s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:46:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.703315161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X.............XXX..........XX......X...............] duration: 5.867334794s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.867356876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.704025885s trace: https://jaeger.dev.storj.io/trace/645ef7eb2c1160f2 FAIL (2025-12-08 07:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:43:53.832679009 (64h14m53.703762054s) [upload=56322]: Starting upload Dec 8 07:43:53.832761265 (64h14m53.70384432s) [upload=56322]: Done starting upload Dec 8 07:43:53.832831138 (64h14m53.703914173s) [upload=56322]: Uploading segments... Dec 8 07:43:53.833040236 (64h14m53.704123272s) [upload=56322]: Got next segment. Inline: false Dec 8 07:43:53.833084621 (64h14m53.704167656s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:43:53.833090904 (64h14m53.704173949s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:44:00.088534242 (64h14m59.959617277s) [seg_pos={0 0}, upload=56322]: Begin upload segment... Dec 8 07:44:00.088651775 (64h14m59.95973482s) [seg_pos={0 0}, upload=56322]: Done begin upload segment. Dec 8 07:44:00.088699727 (64h14m59.959782772s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.088862357 (64h14m59.959945412s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.089914280 (64h14m59.960997315s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090356812 (64h14m59.961439847s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.090841806 (64h14m59.961924851s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091294838 (64h14m59.962377873s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.091928806 (64h14m59.963011851s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093337488 (64h14m59.964420533s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.093575632 (64h14m59.964658667s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095207690 (64h14m59.966290735s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.095281570 (64h14m59.966364615s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.096889231 (64h14m59.967972276s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.100055370 (64h14m59.971138415s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.102060158 (64h14m59.973143203s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.103680393 (64h14m59.974763428s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:00.105348720 (64h14m59.976431755s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Uploading piece... Dec 8 07:44:01.816125519 (64h15m1.687208574s) [upload=56322]: Next returned nil segment Dec 8 07:44:01.816144815 (64h15m1.68722785s) [upload=56322]: Waiting for error group managing segments... Dec 8 07:44:02.114803773 (64h15m1.985886818s) [noise=true , address=145.79.227.27:4689 , piece_id=OT6IG4D65WODDTHRVWW5SAQTRIG6CU3A5JJJ7S6B46DIAANGXT5Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.118905143 (64h15m1.989988188s) [noise=true , address=145.79.227.33:4871 , piece_id=GBI5NEUK3OYAXTFZ7GECXKCQW3MJRJPHE4CEVP63Y3F3EYHDT4UQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.128404291 (64h15m1.999487336s) [noise=true , address=145.79.227.24:4582 , piece_id=XGHZYA7EBLLPGJU32GKOZVLL57QJ2Z2GWV3RLNQZAUAKTICQVG6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.134119694 (64h15m2.005202749s) [noise=true , address=145.79.227.7:4293 , piece_id=6BUKNDYNYL6J4AUCOHGOIABIG24LSL6YFGRIJ6SJKOPPONHBP2QQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.147062700 (64h15m2.018145745s) [noise=true , address=145.79.227.27:4691 , piece_id=U4HCL4L62BL6FK3HGFTK5LFXZ4D67WT7EZOYGEMT7XEHNTUC2LDQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.178455928 (64h15m2.049538973s) [noise=true , address=77.90.31.11:4555 , piece_id=EU3WFLFYPMYGV64HRKOUHFNN5MDRJSYVMERK73FHRHNRN62ZSCFQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.182104134 (64h15m2.053187179s) [noise=true , address=145.79.227.22:4511 , piece_id=ESTKJGABAV4J4U4AWZDV2QJPS62EI35M53AZXGD3CKC6BMMVDCLQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.198945814 (64h15m2.070028869s) [noise=true , address=145.79.227.43:4159 , piece_id=WYFHPLEMW57EBARMW226KUDTNHLJMAPXL5CDIHOF7NKWYP34ITHQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.421237685 (64h15m2.29232073s) [noise=true , address=77.90.31.14:4663 , piece_id=42BO44OTPW5KMCVE3KKYBSAVZGNIA66INRGIMWD3WLJQ646FQ3DQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.560204661 (64h15m2.431287706s) [noise=true , address=145.79.227.28:4727 , piece_id=T73SKSF7KHUKBSISYFH3B24COS6JRZDFXZTXTNPOJHTLCAAOP65Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.592746887 (64h15m2.463829932s) [noise=true , address=145.79.227.26:4653 , piece_id=5JTHJNMRI5UWXNOJUNDMWVOW6J5UV6OWDH34TUTJML7WF3IHRY6Q, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.593195061 (64h15m2.464278106s) [noise=true , address=145.79.227.16:4911 , piece_id=VRHRFUWZ7LXR63LJ7T3G7YG7OVDQ5AYSPR4GAPJKGXMZZNSOWOPA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595002663 (64h15m2.466085708s) [noise=true , address=145.79.227.29:4942 , piece_id=SY7KXG627K3XNWREU7LU6Q44YWNRLXBT6NREORO6BA3MDSU4A7QA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: <nil> Dec 8 07:44:02.595020526 (64h15m2.466103571s) [seg_pos={0 0}, upload=56322]: Segment reached optimal threshold of 13 pieces. Dec 8 07:44:02.595288086 (64h15m2.466371131s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595315549 (64h15m2.466398594s) [noise=true , address=145.79.227.16:4910 , piece_id=HYCFEWFPO6Y3DNTSTSQHPKDRHHGLDWGMYGBH3XAKYADTZOVFP5DA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595441247 (64h15m2.466524292s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: expected piece hash; context canceled; EOF Dec 8 07:44:02.595447961 (64h15m2.466531006s) [noise=true , address=145.79.227.23:4545 , piece_id=C4AJGL6JNUFPPYM2E6LBEZM6MDWFII4NDUV2XEOP2K5AUCOQJFBA, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595457819 (64h15m2.466540864s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->77.90.31.15:4699: use of closed network connection Dec 8 07:44:02.595484661 (64h15m2.466567706s) [noise=true , address=77.90.31.15:4699 , piece_id=7WODLGNN336IYKZ7Y2WKM2KSGESGTXJWLCLH7ODOT5GAGCOCUMNQ, seg_pos={0 0}, upload=56322]: Long tail cancellation due to optimal threshold Dec 8 07:44:02.595501432 (64h15m2.466584477s) [seg_pos={0 0}, upload=56322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:44:02.595610089 (64h15m2.466693134s) [upload=56322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:44:02.595616472 (64h15m2.466699517s) [upload=56322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:44:22.800182125 (64h15m22.67126518s) [upload=56322]: Done uploading segments... Dec 8 07:44:22.800271957 (64h15m22.671355002s) [upload=56322]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.704050642s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 467.406643ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3a746afe6cedeccc FAIL (2025-12-08 07:45:42 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.444424ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.663549872s trace: https://jaeger.dev.storj.io/trace/10deaed4f0583195 FAIL (2025-12-08 07:45:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.663578426s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................................X...X.......XX........] duration: 3.429738395s trace: https://jaeger.dev.storj.io/trace/28e59d7fae0aa278 FAIL (2025-12-08 07:45:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429761628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........X.............XXX..........XX......X..............] duration: 5.816695948s trace: https://jaeger.dev.storj.io/trace/7a866f1583852c11 FAIL (2025-12-08 07:45:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.816899917s core uplink us-select-1 us1-placement-20

1
...
11736
11737
11738
11739
11740
...
17243