Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869232 reports (Page 11618 of 17385)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..................................................X......] duration: 12.261569311s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:29:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 12.261599819s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X........X.....X.......................................X..] duration: 6.725091261s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:29:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.725121117s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 486.594594ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:28:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 486.614372ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.333055991s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:28:52 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.333080387s core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.......................X............XX...X..X............X.] duration: 3.448241935s trace: https://jaeger.dev.storj.io/trace/106afc01b37d11e4 FAIL (2025-12-09 11:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448270308s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..X........X.....X.......................................X.] duration: 6.013053234s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:28:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.01307662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.04951745s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:28:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.04954349s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.05317342s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:28:23 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.053204298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.......................X............XX...X..X............X] duration: 5.001538115s trace: https://jaeger.dev.storj.io/trace/106afc01b37d11e4 FAIL (2025-12-09 11:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001559285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..................................................X.....] duration: 5.815901349s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.815929201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X..X........X.....X.......................................X] duration: 1m0.001895336s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:28:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001917829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X.......................X............XX...X..X............] duration: 3.450224999s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:27:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450250928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..................................................X....] duration: 5.970439501s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:27:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.970470239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.918759292s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:27:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.918845695s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.07967416s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:27:13 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.079709197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................................................X...] duration: 14.141512668s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:26:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.141545621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.937020681s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:26:55 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.937048725s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X..X........X.....X.......................................] duration: 5.596813627s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:26:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.596849166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..................................................X..] duration: 5.672122698s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:26:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.672147926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.712852686s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:26:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.712885256s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X.......................X............XX...X..X...........] duration: 3.551469699s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:26:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551501129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.0240315s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:26:06 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.024065725s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X..X........X.....X......................................] duration: 5.841573104s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:25:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.841602721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 20.624099858s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:25:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 20.624128473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..................................................X.] duration: 9.116654397s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:25:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.116670998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..................................................X] duration: 15m6.762545512s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 15m6.762572093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X..X........X.....X.....................................] duration: 6.010993966s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:25:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.01102247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X...X.......................X............XX...X..X..........] duration: 3.597965468s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:25:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597994634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.950175034s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:24:58 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.950201735s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X..X........X.....X....................................] duration: 5.786326533s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:24:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.786353765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.84968412s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:24:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.849714638s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X...X.......................X............XX...X..X.........] duration: 3.481607628s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:24:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481651111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X...X.......................X............XX...X..X........] duration: 3.405381513s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405407313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X..X........X.....X...................................] duration: 6.128768399s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:23:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.128793205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.96631266s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:23:50 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.966342237s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X...X.......................X............XX...X..X.......] duration: 3.441784859s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:23:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441812611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.381936624s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:23:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.381962283s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.043438315s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:23:21 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.043543706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X...X.......................X............XX...X..X......] duration: 3.484749194s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:23:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484778028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X........X.....X..................................] duration: 5.795450196s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:23:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.795473472s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.......X..X........X.....X.................................] duration: 5.879427469s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:22:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.879498915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.081767082s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:22:14 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.081796359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.048422365s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:22:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.04845686s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.......X..X........X.....X................................] duration: 5.924008657s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:21:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.924033143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X...X.......................X............XX...X..X.....] duration: 3.52781183s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:21:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527838783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.557187594s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:21:48 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.557216157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.881734196s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:21:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.881767038s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X.......X..X........X.....X...............................] duration: 5.875957589s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:21:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.87599472s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X...X.......................X............XX...X..X....] duration: 3.47098989s trace: https://jaeger.dev.storj.io/trace/5e1f67529e67d0c FAIL (2025-12-09 11:21:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471069763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.081266247s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:20:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.081294663s core uplink us-select-1

1
...
11616
11617
11618
11619
11620
...
17385